Subject: Re: [issue76] Comments from Alex

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 6 Jan 2006 13:20:20 +0800

Hi,

The questions below are not editorial ones and have not been solved, so I just
list them again for more discussions.

Question 1. On the State Machine

[Original Text]:

> 3.2.  ForCES Protocol Phases
>    --->
>    ......
>
>
>
>          start                  FEObject Admin up
>          -------+    +--->---->---->---->------->----+
>                 |    |                               Y
>                 Y    |                               |
>                 |    |                               Y
>               +------+--+                        +--------+
>               |         |                        |        |
>               | DOWN    |                        | UP     |
>               | State   |                        | State  |
>               |         |                        |        |
>               +---------+                        +--------+
>                    ^                                  Y
>                    |                                  |
>                    +-<---<------<-----<------<----<---+
>                             FEObject Admin Down/
>                             Association lost
>
>
>    Figure 4: The FE State Machine
>
>    The FE can only be in one of two states as indicated above.  When the
>    FE is in the DOWN state, it is not forwarding packets.  When the FE
>    is in the UP state it may be forwarding packets depending on the
>    configuration of its specific LFBs.
>
>    On start up the FE is in the DOWN state unless explicitly configured
>    by the CE to transition to the UP state.  This MUST be done before
>    configuring any other LFBs that affect packet forwarding.
>
>    The FE transitions from the DOWN to the UP state when explicitly
>    configured to do so by the CE or when it looses its association with
>    the CE.  It should be noted that what needs to be done for the FE to
>    properly complete the transition to the DOWN state is to stop packet
>    forwarding and that this may affect multiple LFBs.  How this is
>    achieved is left as an implementation detail.
>    <---
>

[Comments]:

>       Comments(From Alex): The "State Machine" as aspounced in the segment
below is
>       incomplete in my view. It doesn't address redundancy operation. Is
>       the UP state the same as the ACTIVE state?  Remember, in an ACTIVE/
>       STANDBY operation, the STANDBY element is UP but not ACTIVE.  Also,
>       show the events that cause a transition from STANDBY to ACTIVE and
>       vice-versa.

[Comments from Weiming] I agree that we should refine the 'State Machine'. One
obvious reason is it even confilits with the following text in this same
document:

(06 version Page 47):

      *  CE failover policy - This specifies the behavior of the FE
         during a CE failure and restart time interval, or when the FE
         loses the CE association.  It should be noted that this policy
         in the case of HA only takes effect after total failure to
         connect to a new CE.  A timeout parameter, the CE Timeout
         Interval (CE TI) is associated with this attribute.  Values of
         this policy are defined as below:

            0(default) - The FE should continue running and do what it
            can even without an associated CE.  This basically requires
            that the FE support CE Graceful restart.  Note that if the
            CE still has not been restarted or hasn't been associated
            back to the FE, after the CE TI has expired, the FE will go
            operationally down.

            1 - FE should go down to stop functioning immediately.

            2 - FE should go inactive to temporarily stop functioning.
            If the CE still has not been restarted after a time interval
            of specified by the CE TI, the FE will go down completely.

            Others - Reserved

Comments please!

Thanks,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 6 Jan 2006 14:14:21 +0800

Hi,

Question 2. On the Transaction Operation

This question is raised from our implementation attempt(As Avri mentioned). We
are trying to implement the transaction operation based on the EM and TP flags,
but we just find some questions and unclear points on the usage. We summarize
them in the following text.

Thanks,
Weiming

4.3.1.1.3.  Transaction protocol

   A 2PC [2PCREF] configuration message starts with the header flags
   containing a Start of Transaction (SOT) and Atomic Transaction (AT)
   on its first message configuration message.  A transaction may span
   multiple messages.  It is up to the CE to keep track of the different
   outstanding messages making up a transaction.  As an example, the
   correlator field could be used to mark transactions and a sequence
   field to label the different messages within the same atomic
   transaction.
[Weiming] Obviously, we are missing the definition of the AT(Atomic Transaction)
flag in the draft. I'm not sure if the AT flag is just the 'EM(Excute Mode)'
flag?

   Any failure notified by the FE causes the CE to execute an Abort
   Transaction (ABT) to all FEs involved in the transaction, rolling
   back all previously executed operations in the transaction.

   The transaction commitment phase is signaled from the CE to the FE by
   an empty End of Transaction (EOT) configuration message.
[Weiming]What does it mean by 'empty' here? Is it a config msg that has no
LFBselect? I'm not sure if current BNF grammar allows an config msg without
LFBselect or not?

   The transaction commitment phase is signaled from the CE to the FE by
   an empty End of Transaction (EOT) configuration message.
   The FE MUST respond to the CE's EOT message.  If no response is
   received from the FE within a specified timeout, the transaction MUST
   be aborted by the CE.
[Weiming]We havn't made an attribute to describe the 'timeout' for the
transaction operation here. I suggest we just mention "If no response is
received from the CE, the transaction MUST be aborted by the CE." Another
question here is, what we really mean 'aborted by the CE'? As we know an 'abort
operation' should be done by CE and FE as well. Purely CE abortion will lead to
error in FE.

4.3.1.1.5.  continue-execute-on-failure

   If several independent operations are targeted at one or more LFB
   selectors, execution continues at the FE even if one or more
   operations fail.  This mode is signaled by AT flag set to zero (0).
[Weiming]Again, should it be to 'set the EM flag to 0' rather than the AT flag
to set?

4.3.1.1.7.  Relation to Multipart messages

   Multipart messages, after the first one, are indicated by the Middle
   of Transaction flag (MOT).  The first message is indicated by SOT and
   the last by EOT.

   Multi-part messages MUST be consistent in their use of execution
   modes.  If the first message starts with one exectution mode (such as
   continue-execute-on-failure mode), then all other parts of the same
   multi-part messages MUST continue with the same execution mode.  Any
   inconsistency implies an error and a canceled transaction in which
   all messages are dropped and the sender NACKED.
[Weiming] the NACKED is not clear on its meaning and operation method.

6.1.  Common Header
 .....
       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   |       |   |   |                                         |
       |ACK|  Pri  | EM|TP |           Reserved                      |
       |   |       |   |   |                                         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 .....

       - EM: Execution mode (2 bits)
           There are 3 execution modes refer to Section 4.3.1.1 for
           details.

               Reserved..................... (0b00)

               `execute-all-or-none` ....... (0b01)

               `execute-until-failure` ..... (0b10)

               `continue-execute-on-failure` (0b11)

       - TP: Transaction phase (2 bits)
           A message from the CE to the FE within a transaction could be
           indicative of the different phases the transaction is in.
           Refer to Section 4.3.1.1.3 for details.

               `MOT (Middle of transaction)` (0b00)

               `SOT (start of transaction)` (0b01)

               `EOT (end of transaction)` (0b10)

               `ABT (abort)` (11)

[Weiming]
Q1: is the TP flag only applied to Config message?
Q2: if we now only have one message and we don't use spanning messages
transaction, i.e., we only have one message as one transaction, how should we
set the flag? Just set it to SOT or EOT?
Q3: It seems we now support transactions spanning FEs. Let's show some cases:

Case1. two FEs (FE1 and FE2), with a transaction of 5 messages, msg1 and 2 sent
to FE1 and msg 3,4,5 sent to FE2. How shall we set the TP flag then? Shall we
set the msg1 with SOT msg 2,3,4, with MOT and msg5 with EOT? or shall we set the
msg1 and msg3 with SOT and msg2 and msg 5 with EOT?

Case2. two FEs (FE1 and FE2), with a transaction of 5 messages which should be
sent to FE1 and FE2 via multicast messages. How shall we set the messages then?

Anyway, we are now not very clear by reading the text that, if the SOT, MOT, and
EOT are all only one FE specific or multipul FEs specific.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 10 Jan 2006 08:40:16 -0500

Hi Weiming,

> flag in the draft. I'm not sure if the AT flag is just the 'EM(Excute Mode)'

right - we replaced AT with EM == all-or-none
Can you open an editorial issue tracker on this?

> LFBselect? I'm not sure if current BNF grammar allows an config msg without

The idea was to avoid the LFBSelect altogether since there could be
multiple LFBSelects within a transaction. i.e empty means lacking 
LFBselect.
If the grammar needs resolving, then we should do so - else if the empty
message sounds unreasonable then lets work on alternatives.

> operation' should be done by CE and FE as well. Purely CE abortion will 

Probably the danger in the above text is trying to define how someone
writing code or an app at CE would do two phase commits. Such a person
would have a timeout involved. Maybe we should totally avoid talking
from an app point of view ?

> [Weiming]Again, should it be to 'set the EM flag to 0' rather than the AT flag

indeed. When you open the tracker issue, please create just one to cover
all the AT=> EM translation

>    modes.  If the first message starts with one exectution mode (such as

"exectution" is a typo

>    inconsistency implies an error and a canceled transaction in which

"canceled" should be "cancelled".

> [Weiming] the NACKED is not clear on its meaning and operation method.

This is also one of those editorial issues that need fixing. NACK has
disappeared; above is implying a negative acknowledgement will be
returned as in an error.

> 6.1.  Common Header

It makes sense to be in the config message; i dont see it as being 
harmful in any other message.

> set the flag? Just set it to SOT or EOT?

An atomic transaction must have at least two messages. A start and
end/commit.

> msg1 and msg3 with SOT and msg2 and msg 5 with EOT?

All the rules are based on how 2pc is defined. The danger again is
perhaps in trying to define how someone should code a 2ps.

In this case i would say:

FE1: msg1(SOT), msg2(MOT), msgcommit(EOT)
FE2: msg3(SOT),msg4(MOT),msg5(MOT), msgcommit(EOT)

The "empty" msgcommit is issued to both. A timer is started. If the
timer expires before positive response from all, an ABT is issued to
all. if a response is issued positively from all then the timer is
cancelled.

These are almost implementation issues and the underlying assumption is
that the coder knows what a 2PC is - we just provide the mechanisms.

> sent to FE1 and FE2 via multicast messages. How shall we set the messages then?

The text in the protocol draft should not describe how one would go
implementing 2pc. I think we should probably put a disclaimer saying
so and perhaps in an appendix suggest how this is achieved.

> EOT are all only one FE specific or multipul FEs specific.

I hope the above clarifies things.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 10 Jan 2006 23:10:08 +0800

Jamal,

Very glad to see your comments. (I'd also appreciate it if you can make comments
on the "state machine quesiont" also. thanks.)

I think all the comments have gone to the tracker issue 76, therefore I havn't
tried to open new one for this issue.

To summarize your comments, I'd like to show my understanding of of your idea of
the use of EM and TP flags as below:

1. EM defines if it is a 2PC or not (by setting EM=all-or-none;
2. Then, TP further to define individual messages for the 2PC.
3. When EM is not set to 'all-or-none', the TP is

If it is like this, I do have more question:

1. My former understanding was that EM is not only for inter-messages execution
mode but also for inter-operation TLVs in one message for the execution mode,
i.e., when we set EM=all-or-none, we also mean all operation-TLVs in one message
are also executed in a all-or-none mode. As a result, if we now define EM only
for inter-message execution mode, how we then define intra-message operation
execution mode?

2. If we come back to define that EM is for intra-message as well as for
inter-message, then the question I raised is still there, i.e., when EM is set
to 'all-or-none' and when we only have one message, how should we set the TP
flag?

thanks,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 11 Jan 2006 00:05:15 +0800

Jamal,

When I read once more on the Section 4.3.1 (Transactions, Atomicity, Execution
and Responses), I do believe that a specific flag(1bit) for 2PC should be added.
In this way, the execution as well as 2PC can be unambigious as:

1. EM(2bits) flag  in one message scope only.

2. 2PC(1bit) flag with TP(2bits) flag in multi-messages scopes only.

3. when 2PC is set, the EM MUST be set to 'all-or-none'.

BTW, I also suggest Section 4.3 the subsetion be re-arranged some for
readability. The current subsetion is show as below:

4.3.  Protocol Mechanisms
4.3.1.  Transactions, Atomicity, Execution and Responses
4.3.1.1.  Execution
4.3.1.1.1.  'execute-all-or-none' Atomic transaction
4.3.1.1.2.  Transaction Definition
4.3.1.1.3.  Transaction protocol
4.3.1.1.4.  Recovery
4.3.1.1.5.  continue-execute-on-failure
4.3.1.1.6.  execute-until-failure
4.3.1.1.7.  Relation to Multipart messages
4.3.2.  Heartbeat Mechanism
4.3.3.  FE Object and FE protocol LFBs
4.3.4.  Scalability
4.3.4.1.  Batching
4.3.4.2.  Command Pipelining

I suggest we may re-arrange it as something like:

4.3.  Protocol Mechanisms
4.3.1.  Execution, Transactions, Atomicity, and Responses
4.3.1.1.  Execution
4.3.1.1.1.  execute-all-or-none
4.3.1.1.2.  continue-execute-on-failure
4.3.1.1.3.  execute-until-failure
4.3.1.1. Transaction and Atomicity
4.3.1.1.1.  Transaction Definition
4.3.1.1.2.  Transaction protocol
4.3.1.1.3.  Atomic Transaction
4.3.1.1.4.  Relation to Multipart messages
4.3.1.1.5.  Recovery
4.3.2.  Scalability
4.3.2.1.  Batching
4.3.2.2.  Command Pipelining
4.3.3.  Heartbeat Mechanism
4.3.4.  FE Object and FE protocol LFBs

Just my 2 cent.

Sincerely,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 10 Jan 2006 14:38:39 -0500

> on the "state machine quesiont" also. thanks.)

I will - what is the subject line? I will look for it, but if i dont
respond it is because i havent found it.

> 1. EM defines if it is a 2PC or not (by setting EM=all-or-none;

Well, more like EM defines the execution mode. In which case,
an atomic execution mode (all-or-none) which is used by 2pc is a subset.

> 3. When EM is not set to 'all-or-none', the TP is

TP would be b00 in this case.

> to 'all-or-none' and when we only have one message, how should we set the TP

When we started all this (in the earlier drafts), the EOT and SOT were
two separate bits, in such a scenario one could set the EOT and SOT
together at the same time to mean "this message is the start and end".
Since we have combined them together - the reason being to save on bits
and aggregate the semantice, I would suggest that we need two messages
for atomic transactions (one with a start and another with and end).

If, however, it is perceived that the majority of the messages are going
to be atomic in nature then the extra message is unjustified. We
then could redefine the TP bits appropriately.
This is not a big change in our implementation and i am willing to go
back and make changes depending on consensus.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 10 Jan 2006 14:42:22 -0500

Hi Weiming,

> 3. when 2PC is set, the EM MUST be set to 'all-or-none'.

This is also fine by me as opposed to what I suggested in my previous
email. I would need to think through how it affects our implementation.
The name 2PC doesnt seem fitting. Maybe restore the "AT" flag?

> 4.3.4.  FE Object and FE protocol LFBs

Sounds reasonable..

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 10 Jan 2006 14:49:32 -0500

Hi Weiming,

Is this the message you were refering to?

>             of specified by the CE TI, the FE will go down completely.

The applicability on the HA states (active/backup) is really per _LFB
instance_ NOT per FE as Alex is suggesting. A failover is not
necessarily initiated because a FE is down; as an example a single
instance of a port/link could be failed over from one FE to another.

A failure on a FE or CE OTOH is may mean a failover of all LFB instances
based on the policies you are quoting.

So in summary, I dont see that we need to change that state machine.

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Wed, 11 Jan 2006 11:02:58 +0800

Jamal,

> The name 2PC doesnt seem fitting. Maybe restore the "AT" flag?
I'm ok with the 'AT' flag. I hope this issue can be set asap. How about other
guys' thoughts?

> > 4.3.4.  FE Object and FE protocol LFBs
Jamal, could you provide an update text to Avri so that we can close this whole
issue? Thanks a lot.

cheers,
weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 11 Jan 2006 16:22:03 +0800

Jamal,

On the state machine, my question mainly lies in the expression that the
association lost will lead to FE to DOWN state. This is not the scheme that we
currently apply in the draft. I agree with you that the state machine only deals
with the FE as a whole, therefore it can observe FE only with UP and DOWN, but
some expression should be modified so as to make the state machine not conflict
with other part of the protocol draft. Another expression that may lead to
confusion is the text'FEObject Admin up' and the 'FEObject Admin down' in the
figure. This can easily be taken as 'to start FEObject LFB will make FE UP, to
shutdown the FEObject LFB will make FE DOWN'. I don't think this is the case.
The true is FEObject LFB will always be there after FE is electrically started.
The FE UP or FE DOWN state is only managed as an attribute of the FEObject LFB,
which can be configured by CE.

Based on my above views, I suggest the following modifications:

1. modify the figure
         start                  FEObject Admin up

to a figure as:

        FE start            CE configure
         -------+    +--->---->---->---->------->----+
                |    |                               Y
                Y    |                               |
                |    |                               Y
              +------+--+                        +--------+
              | FE      |                        | FE     |
              | DOWN    |                        | UP     |
              | State   |                        | State  |
              |         |                        |        |
              +---------+                        +--------+
                   ^                                  Y
                   |                                  |
                   +-<---<------<-----<------<----<---+
                    CE configure or FE association lost

2. Accordingly make modifications in the followed text

Original:

   The FE transitions from the DOWN to the UP state when explicitly
   configured to do so by the CE or when it loses its association with
   the CE.  For the FE to properly complete the transition to the DOWN
   state it MUST stop packet forwarding and that this may affect
   multiple LFBs.  How this is achieved is outside the scope of this
   specification.

Changed to:

   The FE as a whole can only be in one of two states as indicated above.  When
the FE is in the DOWN state, it is not forwarding packets.  When the FE is in
the UP state it may be forwarding packets depending on the configuration of its
specific LFBs.
   On start up the FE is in the DOWN state unless explicitly configured by the
CE to transition to the UP state.  This MUST be done before configuring any
other LFBs that affect packet forwarding.

   The FE transitions from the UP to the DOWN (Note the original 'from DOWN to
UP' is a typo, I think)state when explicitly configured to do so by the CE. When
FE loses its association with CE, it MAY also go to DOWN state, depending on the
policy CE wants FE to do after FE loses association with the CE. The policy is
called the CE failover policy and is defined in Section 7.2.1. Note that
according to this policy, FE may also no go to DOWN state after association
lost, instead it may just keep packet forwarding so that it keeps FE in UP
state. For the FE to properly complete the transition to the DOWN state it MUST
stop packet forwarding and that this may affect multiple LFBs.  How this is
achieved is outside the scope of this specification.

   CE configures FE states transitions by means of a so-called FEObject LFB,
which is defined in Section 4.3.3 and Section 7.2. FE state is defined as an
attribute of the FEObject LFB, and CE configuration of the attribute exactly
matches CE configuration of the FE state. Note that even in the FE DOWN state,
the FEObject LFB is still active. FEObject LFB will be automatically loaded and
is in an active state right after FE is started. Refer to Section 4.3.3 and
Section 7.2 for more details.

--------

Comments pls!

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 11 Jan 2006 17:13:13 -0500

On Wed, 2006-11-01 at 16:22 +0800, Wang,Weiming wrote:
> association lost will lead to FE to DOWN state.

The text doesnt say what causes the transition. Association lost
effect maybe one such reason after checking what the policy is.
Administrative choice maybe another. If this is not clear it should be.

>  This is not the scheme that we currently apply in the draft. 

I dont see a conflict - If the text is not clear it needs to be fixed.

> I agree with you that the state machine only deals

Yes, this is true.

> Based on my above views, I suggest the following modifications:

And while it is true:
- "CE Configure" is insufficient - it doesnt say what is being
configured (original text did)
- "FE Association lost" suffers from the same problem.

> UP' is a typo, I think)

The FE is down by default on startup. Only after the association has
happened can the CE configure it to be up.

> state when explicitly configured to do so by the CE. When

I agree with the general principle of mentioning the relation of the
Admin state to what can force transitions - but i think the text above
may be getting into a lot more details on the FEObject LFB than is
necessary.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Thu, 12 Jan 2006 11:28:11 +0800

> From: "Jamal Hadi Salim" <[email-masked]>

> I dont see a conflict - If the text is not clear it needs to be fixed.

The original text has a strong feeling that association lost will lead to FE
DOWN.

> configured (original text did)
original text has a feeling that it is to let FEObject LFB up to let the FE UP
and let FEObject LFB down to let FE DOWN. Another is theres no FEObject
definition before the figure in the draft, hense if you use the original text,
you should mention the FEObject LFB in the followed text. Note that the original
text does not mention FEObject anymore except in the figure.

> - "FE Association lost" suffers from the same problem.
Theres some, but it depends on the text followed to explain the figure.

> happened can the CE configure it to be up.
Yes, the text does not conflict with this.

> may be getting into a lot more details on the FEObject LFB than is
I think it is necessary, or else, it is hard for readers to understand how this
state machine matches the actual operations the draft provides (many of my
implementation team members have asked about this question). But, anyway, please
provide your proposals for the text modification then.

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 12 Jan 2006 09:45:03 -0500

> The original text has a strong feeling that association lost will lead to FE

We need to fix it then.

[other in-context text deleted here ..]

>  But, anyway, please provide your proposals for the text modification then.
> 

I dont have time Weiming. I may disappear again for another 2-3 weeks
and that doesnt work well for the drafts progress. You understand where
i stand - propose the compromising text and pass it on to Avri and if i
have any issues with it i will raise them in the future.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 13 Jan 2006 13:24:18 +0800

Jamal,

> have any issues with it i will raise them in the future.
The text I provided is my proposed text for the update, but it seems you are not
satisfied with the text generally. I can only do modification of the text based
on your comments, I cannot provide a new one for the whole modification. If you
are generally not satisfied with the text I provided last time, I suggest you
provide a new one for this, which may not take you too much time.

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 13 Jan 2006 09:47:37 -0500

On Fri, 2006-13-01 at 13:24 +0800, Wang,Weiming wrote:

> on your comments, I cannot provide a new one for the whole modification.

Thats fine Weiming. Since you feel strongly about the current text, go
ahead and make the changes and before final call i will make sure to
review. I simply dont have time at the moment. 

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Fri, 13 Jan 2006 23:34:25 +0800

Jamal,

> review. I simply dont have time at the moment.
That's ok then.
How about the update for Section 4.3 based on our previous discussions then? I
can try to make it but I have to take some time. I do hope you can do some of it
before you disappear for a moment. Note the time even for the 63th meeting is
closing, I'm just a little worry.

Thanks,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 17 Jan 2006 14:43:57 +0800

Hi,

When discussing the FE state machine, I checked the FE model-05 for the FEObject
definition and found that the 'FEState' has been defined as 'read-only', as:

                <attribute access="read-only" elementID="7">
                   <name>FEState</name>
                   <synopsis>model of this FE</synopsis>
                   <typeRef>FEStatusValues</typeRef>
                 </attribute>

And there are three FEstatusvalues defined as 'AdminDisable', 'OperDisable', and
'OperEnable'.

To make the consistency, I think that the state attribute should be
'read-write', or a new 'read-write' attribute should be additively defined for
'FEState' (the original attribute can be called 'FEStatus'). And I also ask some
explanation on the difference of 'AdminDisable' and 'OperDisable'. Thanks a lot.

Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 17 Jan 2006 17:49:09 +0800

Avri,

Please incorparate the following text changes if theres no further comments.

Thanks,
Weiming

<Original text>

4.2.  ForCES Protocol Phases

   ForCES, in relation to NEs, involves two phases: the Pre-Association
   phase where configuration/initialization/bootup of the TML and PL
   layer happens, and the association phase where the ForCES protocol
   operates to manipulate the parameters of the FEs.

         start                  FEObject Admin up

</original text>

<changed to>

4.2.  ForCES Protocol Phases
   ForCES, in relation to NEs, involves two phases: the Pre-Association
   phase where configuration/initialization/bootup of the TML and PL
   layer happens, and the Post-association phase where the ForCES protocol
   operates to manipulate the parameters of the FEs.

        FE start            CE configures
         -------+    +--->---->---->---->------->----+
                |    |                               Y
                Y    |                               |
                |    |                               Y
              +------+--+                        +--------+
              | FE      |                        | FE     |
              | DOWN    |                        | UP     |
              | State   |                        | State  |
              |         |                        |        |
              +---------+                        +--------+
                   ^                                  Y
                   |                                  |
                   +-<---<------<-----<------<----<---+
                    CE configures or FE loses association

The FE can only be in one of two states as indicated above.  When
the FE is in the DOWN state, it is not forwarding packets.  When the FE is in
the UP state it may be forwarding packets depending on the configuration of its
specific LFBs.

CE configures FE states transitions by means of a so-called FEObject LFB,
which is defined in [FE-MODEL] and also explained in Section 4.3.3 of this
document.
In FEObject LFB, FE state is defined as an attribute of the LFB, and CE
configuration of the FE state
equals CE configuration of this attribute. Note that even in the FE DOWN state,
the FEObject LFB itself is active.

On start up the FE is in the DOWN state unless explicitly configured by the
CE to transition to the UP state.  This must be done before configuring any
other LFBs that affect packet forwarding.

The FE transitions from the UP to the DOWN state when explicitly configured to
do so by the CE or when
FE loses its association with CE. Note that in the association lost case, FE can
also possibly not go to DOWN state. Whether or not FE goes to DOWN state in this
case depends on the policy CE sets for the FE to do when such failover happens.
See  the 'CE failover policy' part in Section 7.2.1 for more details on this.

For the FE to properly complete the transition to the DOWN state it must
stop packet forwarding and that this may affect multiple LFBs.  How this is
achieved is outside the scope of this specification.

</changed to>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 17 Jan 2006 07:46:39 -0500

Sounds like a good idea to me.
I will put it on the list of things to do.
Joel

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 18 Jan 2006 16:06:13 +0800

Avri and all,

I made the update as below. Please make your comments.

Avri, please help to fit in if theres no further comments.

thanks,
Weiming

--------- Section 4.3 updates as below:

4.3.  Protocol Mechanisms

   Various semantics are exposed to the protocol users via the PL header
   including: Transaction capabilities, atomicity of transactions, two
   phase commits, batching/parallelization, High Availability and
   failover as well as command windows. EM (Execute Mode) flag, AT (Atomic
Transaction) flag, and TP (Transaction Phase) flag defined in Common Header
Section (Section 6.1) are highly related to these mechanisms.

   In the master-slave relationship the CE instructs one or more FEs on
   how to execute operations and how to report back the results.

   This section details the different modes of execution that a CE can
   order the FE(s) to perform as defined in Section 4.3.1.1.  It also
   describes the different modes a CE can ask the FE(s) to use for
   formatting the responses after processing the operations as
   requested, which relates to transactional two phase commitment operations.

4.3.1.1.  Execution

   There are 3 execution modes that could be requested for a batch of
   operations spanning one or more LFB selectors in one protocol
   message. The EM flag defined in Common Header Section (Section 6.1) selects
the execution mode for a protocol message, as below:

   a.  execute-all-or-none

   b.  execute-until-failure

   c.  continue-execute-on-failure

   When set to this mode, indipendent operations in a message targeted at one or
more LFB selectors will all be executed if no failure happens for the
operations, or none of the operations will be executed if there is any failure
for the operations.

   operations fail.

   In this mode all operations are executed on the FE sequentially until
   the first failure.  The rest of the operations are not executed but
   operations already completed are not undone, i.e. there is no roll
   back in this mode.

4.3.1.2. Transaction and Atomicity

4.3.1.2.1.  Transaction Definition

   A transaction is defined as a collection of one or more ForCES
   operations within one or more PL messages that MUST meet the ACIDity
   properties[ACID], defined as:

   o  *Atomicity*.  In a transaction involving two or more discrete
      pieces of information, either all of the pieces are committed or
      none are.

   o  *Consistency*.  A transaction either creates a new and valid state
      of data, or, if any failure occurs, returns all data to its state
      before the transaction was started.

   o  *Isolation*.  A transaction in process and not yet committed must
      remain isolated from any other transaction.

   o  *Durability*.  Committed data is saved by the system such that,
      even in the event of a failure and system restart, the data is
      available in its correct state.

   There are cases where the CE knows exact memory and implementation
   details of the FE such as in the case of an FE-CE pair from the same
   vendor where the FE-CE pair is tightly coupled.  In such a case, the
   transactional operations may be simplified further by extra
   computation at the CE.  This view is not discussed further other than
   to mention that it is not disallowed.

   Defined as above, a transaction is always atomic and may be:

   a.  Within an FE alone
       Example: updating multiple tables which are dependent on each
       other.  If updating one fails, then any others already updated
       must be undone.

   b.  Distributed across the NE
       Example: updating table(s) that are inter-dependent across
       several FEs (such as L3 forwarding related tables).

 4.3.1.2.2. Transaction protocol

   By use of the execute mode as defined in Section 4.3.1.1, the protocol has
provided a mechanism for transactional operations within one stand-alone
message. The 'execute-all-or-none' mode can
meet the ACID requirements.

   For transactional operations of multipul messages within one FE or across
FEs, a classical transactional protocol known as Two Phase Commit (2PC) [2PCREF]
is supported by the protocol to achieve the transactional operations.

   AT flag and TP flag in Common Header (Section 6.1) are provided for 2PC based
transactional operations spaaning multipul messages.

   The AT flag, when set, indicates this message belongs to an Atomic
Transaction. All messages for a transaction operation must have the AT flag set.
If not set, it means the message is just stand-alone message and does not
participate in any transaction operation that spans mulitpul messages.

   The TP flag indicates the Transaction Phase this message belongs to. There
are possibly 4 phases for an transactional operation as: `SOT (Start of
Transaction)` , `MOT (Middle of Transaction)`, `EOT (End of Transaction)`, and
`ABT (Abort)`.

   A transaction operation is started with a message the TP flag is set to Start
of Transaction (SOT). Multipart messages, after the first one, are indicated by
the Middle of Transaction flag (MOT).  The last message is indicated by by EOT.

   The transaction commitment phase is signaled from the CE to the FE by
   an End of Transaction (EOT) configuration message.
   The FE MUST respond to the CE's EOT message.  If no response is
   received from the FE within a specified timeout, the transaction MUST
   be aborted by the CE.

Note that a transactional operation is generically atomic, therefore it requires
that the execute modes of all messages in a transaction operation should always
be kept the same and be set to 'execute-all-or-none'. If the EM flag is set to
other execute modes, it will result in a transaction failure.

   As noted above, a transaction may span multiple messages.  It is up to the CE
to keep track of the different outstanding messages making up a transaction.  As
an example, the correlator field could be used to mark transactions and a
sequence field to label the different messages within the same atomic
transaction, but this is out of scope and up to implementations.

4.3.1.2.3.  Recovery

   Any of the participating FEs, or the CE, or the associations between
   them, may fail after the EOT response message has left the FE but
   before it has received all the responses, e.g if the EOT response
   never reaches the CE.

   In this protocol revision, for sake of simplicity as indicated in
   Section 4.2.2.3, an FE losing an association would be required to get
   new state from the newly associated CE from scratch upon a re-
   association.  The decision on what an FE should do after a lost
   association is dictated by the CE Failover policy (refer to Section 9
   and Section 7.2).

4.3.2.  Scalability

   It is desirable that the PL layer not become the bottleneck when
   larger bandwidth pipes become available.  To pick a hypothetical
   example in today's terms, if a 100Gbps pipe is available and there is
   sufficient work then the PL layer should be able to take advantage of
   this and use all of the 100Gbps pipe.  Two mechanisms have actually been
provided to achieve this.  The first one is batching and the second one is a
command window.

   Batching is the ability to send multiple commands (such as Config) in
   one Protocol Data Unit (PDU).  The size of the batch will be affected
   by, amongst other things, the path MTU.  The commands may be part of
   the same transaction or part of unrelated transactions that are
   independent of each other.

   Command windowing allows for pipelining of independent transactions
   which do not affect each other.  Each independent transaction could
   consist of one or more batches.

4.3.2.1.  Batching

   There are several batching levels at different protocol hierarchies.

   o  multiple PL PDUs can be aggregated under one TML message

   o  multiple LFB classes and instances (as indicated in the LFB
      selector) can be addressed within one PL PDU

   o  Multiple operations can be addressed to a single LFB class and
      instance

   The protocol allows any number of messages to be issued by the CE
   before the corresponding acknowledgments (if requested) have been
   returned by the FE.  Hence pipelining is inherently supported by the
   protocol.  Matching responses with requests messages can be done
   using the correlator field in the message header.

   Heartbeats (HB) between FEs and CEs are traffic sensitive.  A HB is
   sent only if no PL traffic is sent between the CE and FE within a
   configured interval.  This has the effect of reducing the amount of
   HB traffic in the case of busy PL periods.

   A HB can be sourced by either the CE or FE.  When sourced by the CE,
   a response can be requested (similar to the ICMP ping protocol).  The
   FE can only generate HBs in the case of being configured to do so by
   the CE.  Refer to Section 7.2.1 and Section 7.9 for details.

   All PL messages operate on LFB constructs as this provides more
   flexibility for future enhancements.  This means that maintenance and
   configurability of FEs, NE, as well as the ForCES protocol itself
   must be expressed in terms of this LFB architecture.  For this reason
   special LFBs are created to accommodate this need.

   In addition, this shows how the ForCES protocol itself can be
   controlled by the very same type of structures (LFBs) it uses to
   control functions such as IP forwarding, filtering, etc.

   To achieve this, the following specialized LFBs are introduced:

   o  FE Protocol LFB which is used to control the ForCES protocol.

   o  FE Object LFB which is used to controls attributes relative to the
      FE itself.  Such attributes include FEState (refer to model
      draft), vendor, etc.

   These LFBs are detailed in Section 7.2.

------- some more changes as below:

Page 28 changed to:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   |       |   |   | |   |                                   |
      |ACK|  Pri  |X X|TP |A|TP |     Reserved                      |
      |   |       |   |   |T|   |                                   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       - ACK: ACK indicator(2 bit)
           The ACK indicator flag is only used by the CE when sending a
           Config Message(Section 7.5.1) or a HB message (Section 7.9)
           to indicate to the message receiver whether or not a
           response is required by the sender.  Note that for all other
           messages this flag MUST be ignored.

(note the change in the text above)

Page 30 changed to:

       - EM: Execution mode (2 bits)
           There are 3 execution modes, refer to Section 4.3.1.1 for
           details.

                Reserved.................... (0b00)

       - AT: Atomic Transaction (1 bit)
This flag indicates if the message is stand-alone message or one of multipul
messages that belongs to 2PC transaction operations. See Section 4.3.1.2 for
details.

               'Stand-alone message'........ (0b0)

               `2PC transaction message' ... (0b1)

       - TP: Transaction phase (2 bits)
           A message from the CE to the FE within a transaction could be
           indicative of the different phases the transaction is in.
           Refer to Section 4.3.1.2 for details.

               `SOT (Start of Transaction)` --- (0b00)

               `MOT (Middle of Transaction)` -- (0b01)

               `EOT (End of Transaction)` ----- (0b10)

               `ABT (Abort)` ------------------ (0b11)

---

From: "Wang,Weiming" <[email-masked]>
Date: Thu, 19 Jan 2006 10:22:15 +0800

Sorry, a typo I found by myself.

> Page 28 changed to:

Should be:

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   |       |   |   | |   |                                   |
       |ACK|  Pri  |x x|EM |A|TP |     Reserved                      |
       |   |       |   |   |T|   |                                   |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Avri, I just found the tracker does not quote issue 76 from Forces list anymore,
which may have to make your editing based on this list directly.

Thanks,
Weiming