Subject: First draft of ForCES protocol from the protocol design team

From: [email-masked]
Date: Wed, 23 Jun 2004 00:26:39 -0400

Hi,

I have just sent our first draft of a "ForCES Protocol Specification" 
to the Internet Drafts editor.

In the meantime, it can be found at:

http://psg.com/~avri/forces/draft-fpteam-forces-protocol-00.txt

a.

For the ForCES protocol design team:
       Ligang Dong
       Avri Doria
       Ram Gopal
       Robert Haas
       Jamal Hadi Salim
       Hormuzd M Khosravi
       Weiming Wang

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 24 Jun 2004 11:15:17 -0400

In reading the protocol proposal (my thanks to the design team for all
their effort) I was struck by an minor terminology question.

We define an NE as a single point of management which contains one or more
CEs and one or more FEs.

We do not define the span of control of a CE.  That is, we do not declare
that all of the FEs in an NE must work with the same CE (with others as
standby).  Nor do we define that all the FEs a CE controls must be within
the NE.  I do not think we intended that the CE span of control always
exactly equals the NE.  We may (or may not) have intended that the CE span
of control is a subset of the NE.

If people think that by NE we have always meant "the span of control of the
CE" then we need to get that written down, and readers can ignore the rest
of this note where I try to explain why I think that is what we should mean.

But for most (maybe even all) ForCES protocol purposes, the only
interesting scope is the scope of control of is the scope over which a CE
has control.  For example, the protocol document provides a very nice
description of some atomicity, transaction commitment, and rollback issues
(ACID).  The text then says that the transaction start and transaction end
bits (probably with a transaction abort message) are sufficient to provide
FE level and NE level ACID properties.  However, the ForCES protocol as far
as I can tell does not actually work at the NE level.  It works at the "CE
span of control" level.  In terms of the ForCES protocol, the bits can only
provide ACID at that level.  Fortunately, that is what I think we need, as
transaction consistency across CEs is clearly outside of our scope.

So, my only point is that the document use "across the NE" because that is
the span we have tended to talk about.  I think we usually mean "across of
span of control of the CE".   One possibility is to explicitly make the NE
equal to the span of control of the CE.

Yours,
Joel

---

From: Alistair Munro <[email-masked]>
Date: Thu, 24 Jun 2004 16:53:10 +0100

Hi Joel,

You are speaking on a topic close to my heart.

I think the scope of administration of FEs by a CE is a policy issue. The
protocol will provide a language in which any CE will be able to speak to any
FE in competition with other CEs. The protocol must provide the mechanisms to
deny access to ineligible CEs (according to local security and policy regimes)
and ensure ACID properties for concurrent access by eligible CEs. What the
messages actually say is for administrators to decide and the language should
be expressive enough (e.g by referring to other status as established by RADIUS
(an example, not a mandate)) to discriminate administrative rights and
priorities in resolving conflicts.

Regards,

Alistair

--
Alistair Munro,
Principal Engineer, U4EA Technologies Ltd.
City Point, Temple Gate, Bristol, BS1 6PL, U.K.
E-mail: [email-masked]
Tel: (work) +44-117-373-6765, (home) +44-1275-462707, (mobile)
+44-7974-922-442;
Fax: +44-117-373-6751
Web: http://www.u4eatech.com

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 24 Jun 2004 14:55:59 -0400

> We define an NE as a single point of management which contains one or more

Thats the general feeling.

> of control is a subset of the NE.

If i understood you correctly, there are two intersecting scopes:
1) Control of a NE (Forces scope) by a CE(s). Such a NE may spun multi
boxes, racks or geographical locations etc
2) Control of network or set of related NEs (maybe within an
administrative domain);

#2 is the superset. Of course this applies as well to things like VRs
within same physical box/chasis.
There should be no problem having the physical CE(s) doing #2 also doing
#1 - call it a super-CE for being knowledgeable about multiple NEs. And
there should be no issues with CE(s) controlling multiple NEs and
residing on the same physical hardware coordinating their activities in
policy making (hence achieveing a single point of management).
The bigger question is whether Forces is the protocol to use for #2 or
whether some other "management"/ "provisioning" protocol (netconf or
SNMP for example) is more appropriate...
Forces fits and can be used in both but it is cleaner to separate the
two scopes.

> of this note where I try to explain why I think that is what we should mean.

Your other text is valid, but i think we should limit the NE to be
within the "the span of control of the CE" (for now at least). If it
needs to be added we should (should probably have happened on the
Framework doc).

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 24 Jun 2004 15:33:51 -0400

Some minor comments on the document.

1) The common header defines a version number field.  It further splits
that version number field into two subfields (major and minor).  I presume
this is intended for some form of negotiation / compatibility.  I further
presume that the absence of a description is due to a lack of
time.  However, if you are going to be this specific about the structure,
some text about the reason for the distinction and what might be affected
by the internal structure would be very helpful.

2) The document frequently says ~the body of message foo consists of one
TLV which looks like~ and then proceeds to give the format and description
of the TLV.  It would seem a good practice if each TLV were given a name
(even if that name is the same as the type in some cases) so I could refer
to the "heart beat interval TLV" or the "result code tlv"  (which should
only be the name if that tlv can appear in multiple different messages as
the result.  Otherwise it should be named "association result TLV".)

3) In the definition of the "heartbeat interval" it would be a good idea if
this description said either ~the interval at which the sender expects to
generate heartbeat messages~ or ~the interval at which the sender requests
the receiver to generate heartbeat messages~.  Either is useful.  Simply
"heartbeat interval" is unclear.

4) It seems wrong to say "the ACK flag in the header is always ignored" in
various message description.  We ought to specify how the flag should be set.
4') We ought to define the meaning of the ack bit.  Choices I can think of:
4'a) The receiver is expected to send a "ack" message when this is
received.  With this meaning, messages with mandatory response messages
would set the bit to one.
4'b) The receiver is expected to send the message-type-specific response
message when this message is received.  For messages with mandatory
responses, the bit would be set to 1.  For messages with optional
responses, the bit would be set according to the intent of the message sender.

5) I think that trying to mix LFB selection with the FE / CE identification
in the last part of bullet G of section 5 is a mistake.  This level of
addressing ought to indicate which FEs I want to talk about.  If I need to
be able to indicate a specific subset of LFBs on FE 1 and a distinct subset
of FE2 while sending both FEs the same message, then we should add
mechanisms to create LFB groupings, and put that group identifier in an LFB
group TLV.

Thank you,
Joel

---

From: "T. Sridhar" <[email-masked]>
Date: Thu, 24 Jun 2004 16:38:54 -0700

Hi,

Here's my set of comments (mostly minor):

1. Section 3.1.1, near the last line - parameterization is spelt
incorrectly.

2. Section 3.2.2.1 - suggest "More details are provided in the section on
protocol scenarios".

3. Section 3.3.1 - second paragraph - it is not clear whether message batch
reassembly needs to take place before exectution. How does this work? Also,
the concept of batch is introduced only in Section 3.3.3, hence the text is
a little difficult to understand in this section.

4. Section 6.2 - the bullet items under "FE (coarse layer) information". The
second bullet states "Inter-FE topology, that is, LFB topology" - shouldn't
that be "Intra-FE topology"?

5. Section 6.5 - A nit - since Packet Redirect messages also involve CE-FE
messages (which is not really a redirect) , should we look at calling it
something else, e.g. Packet Transfer message?

Thanks,
Sridhar

>http://psg.com/~avri/forces/draft-fpteam-forces-protocol-00.txt

***************************************************************************
This message is proprietary to FutureSoft and is intended solely for the use of the individual to whom it is addressed. It may contain  privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended.

If you have received this message in error, please notify the
originator immediately. If you are not the intended recipient,
you are notified that you are strictly prohibited from using,
copying, altering, or disclosing the contents of this message.
FutureSoft accepts no responsibility for loss or damage arising from
the use of the information transmitted by this email including
damage from virus.
***************************************************************************

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sat, 26 Jun 2004 07:12:45 -0400

> 2. Section 3.2.2.1 - suggest "More details are provided in the section on

Noted.

> reassembly needs to take place before exectution.

Suggest some wording. The atomic unit is _always_  a transaction. A
transaction may consist of several operations in a serial order. A
transaction reassembly will be needed if these operations are split
across multiple packets. The ordering/serialization of the operations is
more important.

> How does this work?

Consider the transaction: "route-add A B,C:route-del X"; the FE must
execute the 4 operations in the order specified.
Theres actually a start of transaction indicator, followed by the
operations (add/del) and terminated by a end of transaction indicator.
Scenarios are:
1) the operations + start and end could be sent in one message in such a
case no reassembly needed.
2) The operations + start in first packet and commit in the second.
No reassembly needed.
3) The start + first operation, followed by the 3 operations and the end
are all in separate packets. A reassembly is needed. Note depending on
the TML, the middle operations could arrive out of order; the reassembly
would ensure the ordering.

Attached is some text that is missing from the draft that now that you
ask it makes sense to put into section 3.3.3.1
The diagrams dont show more than one one config operation since the
focus is on the 2pc. If you think it would help to show multiple config
operations let us know.

> Also,

Unfortunately batching applies to both concurency and as a scheme for
unit-izing(if such a word exists) transactions. Maybe we should move the
concurency section 3.3.3 to be 3.3.1 and shift the others down?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 26 Jun 2004 13:09:24 -0400

Conceptually, I can understand wanting two phase commit.  It looks like it
should fit naturally in the protocol.

It would seem that we should have a bit more motivation in the
example.  Normally, two phase commit is used in a situation where having
half the state would be harmful.  With RSVP, it is not clear that there is
any damage done by having only part of the state estalbished (RSVP is
designed to be quite resilient in the face of state loss.)
Also, two-phase commit is particularly relevant when it is difficult to
roll-back the activity after completion.  Typically, such problems occur
due to interaction with outside observers using the result of the action
before it can be unwound.  That does not appear to be an issue in the RSVP
example.

I expect that transaction support over more than a single message, with
explicit commit / abort indications, is probably a good idea.  But it would
be easier to tell if we had a better example.

Yours,
Joel M. Halpern

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 27 Jun 2004 11:09:47 -0400

On Sat, 2004-06-26 at 13:09, Joel M. Halpern wrote:
[..]
> be easier to tell if we had a better example.

Joel,

The mention of QoS and RSVP weakens that text and may even
be a little misleading.

Would the following scenario help as a good example?

Consider a V4 forwarding decision that happens on ingress of FE1 for a
packet coming into port A (of FE1). The lookup decision is to egress on
FE2 port B. This is a very common setup (with FEs interconnected via
some swithc fabric).

Each FEx keeps a local V4 forwarding + nexthop details. In such a case
both FE1 and FE2 need to sync i.e have identical details on
where the nexthop is - which is achieved by having an atomic update of
their respective LFB tables. In other words, should FE1 be pointing to
anything else other than FE2 port B for the egress port (as shown in
above example), then chaos would ensue.

I could draw a diagram to illustrate it if the example doesnt make
sense.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 27 Jun 2004 11:41:09 -0400

It might be possible to elaborate the forwarding table case.
The difficulty is that even here, one can easily see a way around the ACID
requirement.

Suppose that the example were elaborated:
Suppose that the CE wishes to install a new forwarding path for destination
D through FE2.  Given that FE1 has a working forwarding path for
destination D, the CE wants to ensure that the new path is not used by FE1
unless the install is successful in FE2 as well.

However, the reader may realize that this could have been solved without
ACID, just by establishing the FE2 state before the FE1 state.  ACID tends
to apply when there are interactions with other activities.  Either with
other control activiites, or effects on the data path that can not be
cleanly applied in order.

The most common cases I have seen are situations where dtwo objects have to
be created / updated together because they refer to each other.
Let me try another example.
Suppose one were modifying some traffic treatment using a non-work
conserving scheduler.
You start by creating a new queue, and by creating a new classify entry,
which still points to the existing treatment.
Then one goes to create the non-work conserving handling.  As soon as the
CE creates the scheduling entry, there is an effect on other traffic
behavior even if there is nothing in the queue feeding this new entry.
At the same time, as soon as the CE adjusts the port selection of the
classifier, traffic will be diverted to the queue.
If either action occurs without the other, undesired results occur.
Thus, one wants ACID at least with a transaction on an FE.
And of course the two pieces could have been in different FEs.

I believe this is what was behind the selection of QoS / RSVP as an
example.  But we need the elaboration to explain why it matters.
(Admittedly, non-work-conserving schedulers are not common.  But they do
occur.  And the real point is to show the potential for interaction that
can not be safely serialized.)

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 27 Jun 2004 15:05:26 -0400

On Sun, 2004-06-27 at 11:41, Joel M. Halpern wrote:
> ACID, just by establishing the FE2 state before the FE1 state.

Assuming the path to destination D on packets coming into FE1 was
pointing to say FE20 port E - there is a possibility of an ambiguity.
I.e you have a window of time where packets coming in via FE1 will
continue going through FE20 port E while packets coming into FE2 to
destination D now go through the (new) correct path.
Unless such races are tolerable - which is a tradeoff if the ambiguities
exist in limited timescales or a CE policy intent.

> ACID tends to apply when there are interactions with other activities.
> Either with other control activiites, or effects on the data path that
> can not be cleanly applied in order.

nod.

> be created / updated together because they refer to each other.

Refer to each other or would the case of one refer to other also be
a sufficient condition?

> behavior even if there is nothing in the queue feeding this new entry.

Good example.
You picked an example that requires a resource to be
reserved/preallocated --> implication is that such a resource is denied
to other queues once reserved. At least thats what i think you are
getting to.

The first phase of the 2pc is infact a check for availability of a
resource. It asks the question whether a transaction can be achieved.
Show stoppers to a transaction could be lack of resources, conflicts,
etc. At the end of a the first phase, no resource is
preallocated/reserved AFAIK; of course how long such a guarantee is
kept without reservation is probably implementation specific.
Typically, assuming an intelligent CE, there would be no other
transaction requesting for the same resource from the CE until either
the transaction is aborted or timed out.

Back to the forwarding example: Could you say that adjusting tables
infact falls under the same category?

> And of course the two pieces could have been in different FEs.

True.

> can not be safely serialized.)

By definition a transaction (which could be a series of operations) is
atomic.
I believe anything that is affecting state of the NEs distributed
computation needs to be a single transaction (and by definition atomic)
across the NE. Thats why i brought the forwarding example because the
next hop selection may happen on one FE and the L2 header rewrite may
happen on the other. Same concept applies to things that affect a state
within a FE such as the work conserving example you gave.

So maybe a good way is to fit what goes together is by describing
transaction which could be made of a series of operations (refer to
header semantics and the config message).

Thoughts?

cheers,
jamal

---

From: Robert Haas <[email-masked]>
Date: Mon, 28 Jun 2004 17:11:16 +0200

Joel,
Thanks for your comments. See some answers below.

Joel M. Halpern wrote:

> [snip]

I copied the text of the draft you were refering to below:

               It is desirable to address multicast and/or broadcast
               messages to some LFB instances of a given class.  For
               instance, assume FEs FEa and FEb:
               -   FEa has LFBs LFBaX1 and LFBaX2 of class X
               -   similarly, FEb has two LFBs LFBbX1 and LFBbX2 of
                   class X.
               A broadcast message should be addressable to only LFBs
               LFBaX1 and LFBbX1 (this can be the case for instance if
               these two LFBs belong to the same VPN).  To achieve this,
               a VPN ID (3 octets OUI and 4 octets VPN Index) as defined
               in RFC 2685 should be used within the ForCES message body
               as a TLV.

               As an alternative, a particular multicast ID MAY be
               associated to a given VPN ID through some configuration
               means.  Messages delivered to such a multicast ID MUST
               only be applied to LFBs belonging to that VPN ID.

What you call "LFB grouping TLV" is what is proposed here as the VPN ID 
TLV.
A simple mechanism would be to use the broadcast FE ID and add the VPN 
ID TLV in the message so that only those FEs that belong to the VPN 
process the message.
Some people might then suggest that the message should be delivered only 
to those FEs that will process it (to save network bandwidth and CPU 
cycles, for instance), therefore the proposal to use a multicast ID 
instead. Note that the VPN ID TLV could remain in the message, if this 
is what you suggest ?

Regards,

-- 
Robert Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-1-724-8698  fax +41-1-724-8578  http://www.zurich.ibm.com/~rha

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 28 Jun 2004 11:24:17 -0400

Thanks Robert.  I think my problem is only in the english usage that tried 
to treat the multi-target TLV as part of the "message 
addressing".  Mentally, I do not think of the multicast as "addressed" to 
those selective LFBs, but rather as the multicast is addressed to the 
relevant FEs, and the FEs apply the message to the right set of LFBs.
Given that the information is in a TLV and appears to be in a reasonable 
plac ein the message, I don't think we are disagreeing about the structure.

I also just have a mental problem thinking of it as a VPN.
If we need this, could we call it an "LFB Group" instead?  (I do see the 
mental parallels that tempt one to call it a VPN, but I think it is 
actually not a good model.)

Yours,
Joel

PS: It is going to take extra mechanisms to manage these LFB Group IDs.  I 
suppose that the Group_ID -> LFB instance list mapping would be an FE 
Protocol attribute, and the CE would manage the table so that every FE 
knows which of its LFBs are selected.  (I think this belongs in the FE 
Protocol Object rather than in the FE Object.)

At 05:11 PM 6/28/2004 +0200, Robert Haas wrote:
>What you call "LFB grouping TLV" is what is proposed here as the VPN ID TLV.
>A simple mechanism would be to use the broadcast FE ID and add the VPN ID 
>TLV in the message so that only those FEs that belong to the VPN process 
>the message.
>instead. Note that the VPN ID TLV could remain in the message, if this is 
>what you suggest ?
>
>Regards,
>
>--
>Robert Haas

---

From: Robert Haas <[email-masked]>
Date: Wed, 30 Jun 2004 10:45:49 +0200

Joel,

Joel M. Halpern wrote:

> Thanks Robert.  I think my problem is only in the english usage that 
> tried to treat the multi-target TLV as part of the "message 
> addressing".  Mentally, I do not think of the multicast as "addressed" 
> to those selective LFBs, but rather as the multicast is addressed to 
> the relevant FEs, and the FEs apply the message to the right set of LFBs.
> Given that the information is in a TLV and appears to be in a 
> reasonable plac ein the message, I don't think we are disagreeing 
> about the structure.
>
> If we need this, could we call it an "LFB Group" instead?  (I do see 
> the mental parallels that tempt one to call it a VPN, but I think it 
> is actually not a good model.) 

I reused the already defined IETF VPN ID, as the main motivation was to 
have the NE support multiple VPNs, each corresponding to an LFB group.

What is the context that you envision for such LFB groups ?

>
> PS: It is going to take extra mechanisms to manage these LFB Group 
> IDs.  I suppose that the Group_ID -> LFB instance list mapping would 
> be an FE Protocol attribute, and the CE would manage the table so that 
> every FE knows which of its LFBs are selected.  (I think this belongs 
> in the FE Protocol Object rather than in the FE Object.) 

Good point: although it is not mentioned in the draft yet, as you say, 
we can use an 2-dimensional-array attribute in the FE Protocol Object to 
list:
- the groups to which an FE belongs to, and
- the corresponding LFBs in each of those groups.

The mechanisms to manipulate this atrribute will be the Config/Query 
primitives already defined. Or do you see sth missing ?

Regards,
-Robert

>> A simple mechanism would be to use the broadcast FE ID and add the 
>> VPN ID TLV in the message so that only those FEs that belong to the 
>> VPN process the message.
>> Some people might then suggest that the message should be delivered 
>> only to those FEs that will process it (to save network bandwidth and 
>> CPU cycles, for instance), therefore the proposal to use a multicast 
>> ID instead. Note that the VPN ID TLV could remain in the message, if 
>> this is what you suggest ?
>>
>> Regards,
>>
>> -- 
>> Robert Haas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 30 Jun 2004 12:31:03 -0400

The VPN ID space was designed to have IDs from multiple allocators used in 
the same (internet) space.  I think in this case we have only one assigner 
(the CE) within the operational space.  So a simple CE assigned 32 bit 
quantity seems sufficient.  And in general I just have trouble seeing the 
parallel to VPNs.

I agree that the FE table you describe does what is needed.  I was then 
asking whether that belonged in the FE Protocol Object or the FE Attributes 
Object.

Yours,
Joel

At 10:45 AM 6/30/2004 +0200, Robert Haas wrote:
>>IDs.  I suppose that the Group_ID -> LFB instance list mapping would be 
>>an FE Protocol attribute, and the CE would manage the table so that every 
>>FE knows which of its LFBs are selected.  (I think this belongs in the FE 
>Good point: although it is not mentioned in the draft yet, as you say, we 
>can use an 2-dimensional-array attribute in the FE Protocol Object to list: