Subject: draft-audu-forces-pl-00 : Please review and comment.

From: Alex Audu <[email-masked]>
Date: Sat, 17 Jul 2004 16:36:05 EDT

Hi All,
Please take a few moments to review this draft. Comments appreciated.
Thanks,
Alex.
===================================================================
Internet Draft                                        Alex Audu  
Expiration: December 23 2004                      Alcatel USA Inc.    
File: draft-audu-forces-PL-00.txt         
Working Group: ForCES                          

                              June 24 2004 

         Forwarding and Control Element Separation Protocol Layer 
               draft-audu-forces-PL-00.txt 

Status of this Memo

  This document is an Internet-Draft and is in full conformance with 
  all provisions of Section 10 of [STD].  Internet-Drafts are 
  working documents of the Internet Engineering Task Force (IETF), 
  its areas, and its working groups.  Note that other groups may 
  also distribute working documents as Internet-Drafts.

  Internet-Drafts are draft documents valid for a maximum of six 
  months and may be updated, replaced, or obsoleted by other 
  documents at any time.  It is inappropriate to use Internet-Drafts 
  as reference material or to cite them other than as ``work in 
  progress.''

  The list of current Internet-Drafts can be accessed at 
  http://www.ietf.org/ietf/1id-abstracts.txt.

  The list of Internet-Draft Shadow Directories can be accessed at 
  http://www.ietf.org/shadow.html.

Conventions used in this document 

  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 
  "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in 
  this document are to be interpreted as described in [KEYWORDS].

Abstract

  This document defines the Forces-PL protocol that is designed for 
  communicating between Forwarding and Control Elements that make up a 
  ForCEs-compliant network element. This protocol addresses all the 
  requirements described in the Forces [FORCES-REQ] requirements document. 
  This document also specifies architectural attributes necessary in an 
  implementation of Forces-PL to ensure correct and secure protocol operation.

Table of Content

  1. Definitions    . . . . . . . . . . . . . . . . . . . . . . 5
  2. Introduction   . . . . . . . . . . . . . . . . . . . . . . 7
  3. Protocol Overview  . . . . . . . . . . . . . . . . . . . . . . 7
  3.1. Independence from Interconnect   . . . . . . . . . . . . . . 9
  3.2. Reliability  . . . . . . . . . . . . . . . . . . . . . . 9
 audu               Expires December 23, 2004                 Page 1

Internet-Draft                   ForCES-PL               June 24 2004
  3.3. Fail-Over Model  . . . . . . . . . . . . . . . . . . . . . .10
  4. Forces-PL Message Overview . . . . . . . . . . . . . . . . . .10 
  4.1. Protocol Message Header structure    . . . . . . . . . .10
  4.1.1. Version    . . . . . . . . . . . . . . . . . . . . . .11
  4.1.2. eType      . . . . . . . . . . . . . . . . . . . . . . . .11
  4.1.3. Prio (Priority Bits)   . . . . . . . . . . . . . . . . . .11
  4.1.4. Message Classes and Types  . . . . . . . . . . . . . .12
  4.1.5. Length . . . . . . . . . . . . . . . . . . . . . . . . . .14
  4.1.6. Element Tag    . . . . . . . . . . . . . . . . . . . . . .14
  4.1.7. Source Tag . . . . . . . . . . . . . . . . . . . . . .15
  4.1.8. Destination Tag    . . . . . . . . . . . . . . . . . .15
  4.1.9. Transaction Sequence Number (TSN)  . . . . . . . . . .15
  4.2. Service Data or Payload Structure    . . . . . . . . . .15
  5. Forces-PL Messages . . . . . . . . . . . . . . . . . . . . . .17
  5.1. Association and Connection (CA) Messages . . . . . . . . . .17
  5.1.1. Join Request   . . . . . . . . . . . . . . . . . . . . . .17
  5.1.2. Join Response  . . . . . . . . . . . . . . . . . . . . . .18
  5.1.3. Leave Request  . . . . . . . . . . . . . . . . . . . . . .19
  5.1.4. Leave Response . . . . . . . . . . . . . . . . . . . . . .21
  5.1.5. Join Multicast Address (JoinMCastAddr) Request . . . . . .21
  5.1.6. Join Multicast Address Response    . . . . . . . . . .22
  5.1.7. Leave Multicast Address Request    . . . . . . . . . .22
  5.1.8. Leave Multicast Address Response   . . . . . . . . . .23
  5.2. Capabilities Control (CC) Messages   . . . . . . . . . .24
  5.2.1. Capability Request . . . . . . . . . . . . . . . . . .24
  5.2.2. Capability Response    . . . . . . . . . . . . . . . . . .25
  5.2.3. Configure Request  . . . . . . . . . . . . . . . . . .26
  5.2.4. Configure Response . . . . . . . . . . . . . . . . . .27
  5.2.5. Topology Request   . . . . . . . . . . . . . . . . . .28
  5.2.6. Topology Response  . . . . . . . . . . . . . . . . . .29
  5.2.7. Query Request  . . . . . . . . . . . . . . . . . . . . . .29
  5.2.8. Query Response . . . . . . . . . . . . . . . . . . . . . .31
  5.2.9. Error TLV  . . . . . . . . . . . . . . . . . . . . . .31
  5.3. Protocol Element (PE) Maintenance Messages   . . . . . .32
  5.3.1. Protocol Element Up (PEUP) . . . . . . . . . . . . . .32
  5.3.2. Protocol Element Up Acknowledge (PEUP-ACK) . . . . . .32
  5.3.3. Protocol Element Active (PEACT)    . . . . . . . . . .33
  5.3.4. Protocol Element Active Acknowledge (PEACT-ACK)  . . . . .34
  5.3.5. Protocol Element Inactive (PEINACT)    . . . . . . . . . .34
  5.3.6. Protocol Element Inactive Acknowledge (PEINACT-ACK)  . . .35
  5.3.7. Protocol Element Down (PEDOWN) . . . . . . . . . . . . . .35
  5.3.8. Protocol Element Down Acknowledge (PEDOWN-ACK) . . . . . .36
  5.3.9. Heartbeat  . . . . . . . . . . . . . . . . . . . . . .36
  5.3.10. Heartbeat Acknowledge (HB-ACK)    . . . . . . . . . . . .37
  5.4. PE Traffic Maintenance (TM) Messages . . . . . . . . . .37
  5.4.1. Control Packet Redirect to CE From FE  . . . . . . . . . .37
  5.4.2. Control Packet Forwarding to FE from CE    . . . . . . . .38
  5.4.3. Control Packet Forwarding Acknowledgement  . . . . . . . .39
  5.5. Event Notification Messages  . . . . . . . . . . . . . . . .39
  5.5.1. Event Register . . . . . . . . . . . . . . . . . . . . . .39
  5.5.2. Event Register Acknowledgement . . . . . . . . . . . . . .40
  5.5.3. Event De-Register  . . . . . . . . . . . . . . . . . .40
  5.5.4. Event De-Register Acknowledgement  . . . . . . . . . .40
  5.5.5. Asynchronous Event Notification    . . . . . . . . . .40
 audu               Expires December 23, 2004                 Page 2

Internet-Draft                   ForCES-PL               June 24 2004
  5.6. Application And Vendor Specific Function Message Handling. .41
  5.6.1. Application And Vendor Specific Data (AV-DATA Request) . .41
  5.6.2. Vendor Specific Data Ack (VS-Data Ack) . . . . . . . . . .42
  6. Procedures for Forces-PL Protocol  . . . . . . . . . . . . . .42
  6.1. CE and FE State Maintenance  . . . . . . . . . . . . . .42 
  6.1.1. CE and FE States   . . . . . . . . . . . . . . . . . .43
  6.2. State Maintenance Procedures . . . . . . . . . . . . . .44
  6.2.1. Protocol Element Up    . . . . . . . . . . . . . . . . . .44
  6.2.2. Protocol Element Down  . . . . . . . . . . . . . . . . . .45
  6.2.3. Protocol Element ACTIVE    . . . . . . . . . . . . . .45
  6.2.4. Protocol Element Inactive  . . . . . . . . . . . . . .46
  7. Example Scenarios  . . . . . . . . . . . . . . . . . . . . . .46
  7.1. Establishment of Association . . . . . . . . . . . . . .46
  7.2. Steady State Communication   . . . . . . . . . . . . . .47
  7.3. CE Fail-over Scenarios   . . . . . . . . . . . . . . . . . .48   
  8. Security Considerations    . . . . . . . . . . . . . . . . . .50
  8.1. TLS Usage with Forces-PL . . . . . . . . . . . . . . . . . .50
  9. Architecture support for Forces-PL protocol  . . . . . . . . .51
  9.1. Configurable parameters  . . . . . . . . . . . . . . . . . .51
  10. IANA Considerations   . . . . . . . . . . . . . . . . . .51
  11. References    . . . . . . . . . . . . . . . . . . . . . .51
  11.1. Normative References    . . . . . . . . . . . . . . . . . .51
  11.2. Informative References  . . . . . . . . . . . . . . . . . .52
  12. Acknowledgments   . . . . . . . . . . . . . . . . . . . . . .52
  13. Authors' Addresses    . . . . . . . . . . . . . . . . . .52

1. Definitions

  The following definitions are taken from [FORCES-REQ],[FE-MODEL]

  Forwarding Element (FE) - A logical entity that implements the ForCES 
  protocol.  FEs use the underlying hardware to provide per-packet processing 
  and handling as directed by a CE via the ForCES protocol.  FEs may use PFE 
  partitions, whole PFEs, or multiple PFEs.

  Control Element (CE) - A logical entity that implements the ForCES protocol 
  and uses it to instruct one or more FEs how to process packets.  CEs handle 
  functionality such as the execution of control 
  and signaling protocols. 

  Pre-association Phase - The period of time during which a FE Manager (see 
  below) and a CE Manager (see below) are determining which FE and CE should 
  be part of the same network element and delivering that information to the 
  FE and CE.

  Post-association Phase - The period of time during which a FE has the 
  information specifying what CE is to control it and vice versa, including 
  the time during which the CE and FE are establishing communication with 
  one another.

  ForCES Post-Association Phase Protocol - The protocol used for 
  post-association phase communication between CEs and FEs.  This protocol 
  audu               Expires December 23, 2004                 Page 3

Internet-Draft                   ForCES-PL               June 24 2004

  does not apply to CE-to-CE communication, FE-to-FE communication, or to 
  communication between FE and CE managers.  The ForCES protocol is a 
  master-slave protocol in which FEs are slaves and CEs are masters.  This 
  protocol includes both the management of the communication channel (e.g., 
  connection establishment, heartbeats) and the control messages themselves.
  The term ForCES protocol may refer to a suite of protocols that are used 
  to exchange control information as well as redirect data packets between 
  the CEs and FEs.

  FE Manager - A logical entity that operates in the pre-association phase 
  and is responsible for determining with which CE(s) an FE should 
communicate.
  This process is called CE discovery and may involve the FE manager learning 
  the capabilities of available CEs.  A FE manager may use anything from a 
  static configuration to a pre-association phase protocol (see below) to 
  determine which CE(s) to use.  Being a logical entity, an FE manager might 
  be physically combined with any of the other logical entities mentioned in 
  this section.

  CE Manager - A logical entity that operates in the pre-association phase 
  and is responsible for determining with which FE(s) a CE should communicate.
  This process is called FE discovery and may involve the CE manager learning 
  the capabilities of available FEs.  A CE manager may use anything from a 
  static configuration to a pre-association phase protocol (see below) to 
  determine which FE to use.  
  Being a logical entity, a CE manager might be physically combined with any 
  of the other logical entities mentioned in this section.

  Pre-association Phase Protocol - A protocol between FE managers and CE 
  managers that is used to determine which CEs or FEs to use.  A 
  pre-association phase protocol may include a CE and/or FE capability 
  discovery mechanism.  Note that this capability discovery process is wholly 
  separate from (and does not replace) that used within the ForCES protocol.  
  However, the two capability discovery mechanisms may utilize the same FE 
  model.  

  ForCES Network Element (NE) - An entity composed of one or more CEs and one 
  or more FEs.  To entities outside a NE, the NE represents a single point of 
  management.  Similarly, a NE usually hides its internal organization from 
  external entities.

  ForCES Protocol Element (PE) - A Forwarding Element or a Control Element 
  that speaks the ForCES protocol.

  LFB (Logical Function Block) class (or type) -- A generic template 
  representing a fine-grained, logically separable and well-defined packet 
  processing operation in the datapath.  LFB class is the basic building block
  of the FE model.

  FE Model (FEM) -  Modeling/Organization of LFBs in the Forwarding plane.

2. Introduction
 
 audu               Expires December 23, 2004                 Page 4

Internet-Draft                   ForCES-PL               June 24 2004
  Network Elements (NE) such as routers are becoming more and more complex 
  as they try to cope with demanding features like policy based routing, 
  firewalls and NATs, and QoS aware routing. As a result, issues like 
  scalability, (the ability to cost-effectively grow a network as demand 
  increases) and programmability (the ability to dynamically program the 
  network for some specific services by programming the NEs that handle 
  those services) become very important. The ForCEs protocol has been 
  specified to help resolve these issues by decoupling control and forwarding
  parts of a network element, and also adding programmability features to 
  the NE.  

  It is also important for the ForCES protocol to run over varying transport 
  media. To this end, ForCES has been split into two layers: the Protocol 
  Layer (PL) and the Transport Mapping Layer (TML). The PL is generic and 
  common to all implementations of ForCES and is standardized by this IETF 
  document. The TML is defined in other documents. There is a TML generated 
  per transport medium. The default TML is the IP-TML  that corresponds to a 
  TCP/IP transport medium. The (PL) sits on top of (and receives services 
  from) the (TML).

  Forces-PL has been designed for use in a redundant environment, for relaying
  messages between control elements (CE) and forwarding elements (FE) 
  distributed in a network as found in ForCEs. The relationship between CEs 
  and FEs is a master/slave one.

 audu               Expires December 23, 2004                 Page 5

Internet-Draft                   ForCES-PL               June 24 2004

3. Protocol Overview

  ForCES is a framework consisting of set of protocols representing the 
  forwarding and control elements in the form of an extensible model 
  [FRAMEWK],[FE-MODEL].  CEs handle control, signaling and management 
  protocols, while FEs perform forwarding functions. CEs control the behavior
  of FEs in a master/slave fashion.

  To handle the transport of data and control between the CEs and FEs, ForCES 
  has specified two protocol entities: ForCES-Protocol Layer (Froces-PL) and 
  the Transport Mapping Layer (Forces-TML). The ForCES-PL is independent of 
  the transport interconnect type, but it requires service from the ForCES-TML
  to communicate with its peer.
                     
                     +-----------+
                     | Forces-PL |  CE
             |           |
                     +-----------+
                     |   TML     |  Control Plane
                     +-----------+
                      | | | | | |  
           uw1        | | | | | |  
      +---------------+ | | | | +---------------+
      |  +-uw2----------+ | | +--------------+  |
      |  |                | |                |  |
      |  |  +--mw1--------+-|--------------+ |  |
      |  |  |               |              | |  |
      |  |  | +--mw2--------+------------+ | |  |
      |  |  | |                          | | |  |
   +-----------+                        +-----------+
   |   TML     |                        |   TML     |
   +-----------+     . . .   ..         +-----------+  Forwarding
   | Forces-PL |                        | Forces-PL |   Plane
   +-----------+                        +-----------+
       FE1                                    FEn

   Legend:   Forces-PL    - Protocol Layer
        TML   - Transport Mapping Layer
        uwi   - unicast wire with priority i
        uwj       - unicast wire with priority j 
        mwk   - multicast wire with priority k 
        mwl       - multicast wire with priority l 

  Figure 1. Forces-PL and TML connecting FEs and CE

  In Figure 1, virtual wires or links connect a CE in the control plane to a 
  set of FEs (FE1 to FEn)  in the forwardding plane. There are two types of 
  links : unicast and multicast. Unicast links carry unicast data or control 
  between endpoints. Multicast links carry point-to-multipoint data or 
control.
  Each link can be assigned a priority. 

  The links could be of different quality of service (e.g. reliable, or 
  unreliable), but they are all congestion aware. This allows the protocol to 
 audu               Expires December 23, 2004                 Page 6

Internet-Draft                   ForCES-PL               June 24 2004

  separate control and data traffic into different streams, to reduce the 
  threat of Denial of Service (DoS) attacks on the survivability of the system
  and making the protocol more robust.

  In a redundant system, CE s and FE s will be replicated, with one set being
  active at a particular time while the other is in standby.
 
  [DO: summarize main functions of PL here, followed by those of TML]

  The ForCES  protocol also provides a notion of distributed IPC mechanism by 
  providing support functions required for replication, high availability and
  fail-over support for the ForCES distributed network element environment.  

3.1. Independence from Interconnect

  The Forces-PL protocol is independent of the Interconnect Layer. It makes 
  no assumptions about the interconnect layer and uses interconnect 
  independent addressing  in its common header and API.  
  All interconnect specific properties are encapsulated by the TML.

3.2. Reliability

  Separate Control and Data channels

  The ForCES NEs are subject to Denial of Service (DoS) attacks 
  [FORCES-REQ Section 7 #15]. A malicious system in the network can flood a 
  ForCES NE with bogus control packets such as spurious RIP or OSPF packets 
  in an attempt to disrupt the operation of and the communication between the 
  CEs and FEs. In order to protect against this situation, the ForCES protocol
  uses separate control and data channels for communication between the CEs 
  and FEs. 

  The data channel carries the control protocol packets such as RIP, OSPF 
  messages as outlined in [FORCES-REQ] Section 7 #10, which are carried in 
  ForCES-PL PE Traffic Maintenance messages (section 5.4), between the CEs 
  and FEs. All the other Forces-PL messages, which are used for 
  configuration/capability exchanges, event notification, etc, are carried 
  over the control channel. The data channel is set up only after the control 
  channel is set up via the use of the Join process (see section 5.1.1). The 
  CE signals the FE to establish the data channel at the appropriate time. 

  The priority bits in the Forces-PL common header can be used to distinguish 
  between different types of data traffic on the data channel. For example, 
  OSPF packets (encoded as  PE Traffic Maintenance messages) could be given 
  higher priority than ping packets on the data channel. The use of separate 
  control and data channels along with rate limiting mechanisms on the FE 
  provides robustness to the Forces-PL protocol against DoS attacks.

 audu               Expires December 23, 2004                 Page 7

Internet-Draft                   ForCES-PL               June 24 2004

3.3. Fail-Over Model

  The Forces-PL protocol provides CE fail-over functions in order to support 
  high availability of the network element [FORCES-REQ]. 

  The CE-SET (see section 4.1.4) is a list of CEs that reside within a Network
  Element (NE) as a cooperating unit. Likewise, the FE-SET is a list of FEs 
  that reside in an NE as a cooperating unit. 

  The following are the high-availability mechanisms that are provided by 
  Forces-PL protocol. 

  (1) Strong Consistency: In strong consistency mode, the FE sends all 
      asynchronous notifications/ control protocol packets to the primary and 
      backup CEs in the CE set. By doing this, the FE enables both the primary
      and backup CEs to keep the state synchronized.

  (2) Weak Consistency (Fail-over): In this mode, the FE communicates directly
      with the primary CE. If the primary CE fails, the FE starts 
      communicating with the backup CE. The Forces-PL protocol specifies that 
      selection of primary and backup CEs be done during pre-association 
phase.

  In all the above cases, CE (including primary and backup CEs) and FEs are 
  pre-configured to perform such activities as part of pre-association phase. 

4.  Forces-PL Message Overview

  Force-PL protocol messages are made up of two parts: the common header, and 
  the message body or service payload part. This section describes the details
  of the common header and payload structure.

  All messages are sent network Byte Ordere across the network.

4.1.  Protocol Message Header structure

  Forces-PL protocol Header is fixed size and contains the following fields.

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |vers   |eType  | prio| reservd |  msgClass     |   msgType     |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |                       message length                          |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |          Source-Tag           | Destination-Tag               |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        |                        Sequence Number                        |
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 audu               Expires December 23, 2004                 Page 8

Internet-Draft                   ForCES-PL               June 24 2004

4.1.1.  Version

  The version field (4-bits) contains the version of the Forces-PL protocol 
  supported by the implementation. The current supported version is :

     value 0x01

  Protocol elements implementing the Forces-PL protocol SHOULD provide 
  backwards compatibility with prior versions of the protocol.

4.1.2. eType

  This field identifies the format of data exchange used between the 
  communicating endpoints. Valid values include:

  Value   Description
    0x1       TLV
    0x2       XML
    0x3       BINARY-XML

4.1.3. Prio (Priority Bits)

  This three bit field allows eight(8) different levels of priority to be 
  assigned to different packet streams. This enables the different packet 
  streams to be given preferential treatments based on the priority. The 
  default setting is 0 (for normal priority)

4.1.4.  Message Classes and Types

  Forces-PL's messages are grouped into six (6) classes namely: 
  1) Connection and Association messages, which help establish logical 
     connections between FEs and CEs,
  2) Capabilities Control messages, which the CE uses to query and configure 
     the capabilities of the FE,
  3) State Maintenance messages, which are used to track element states,
  4) Traffic Maintenance messages, which are used to exchange control packets 
     between CEs and FEs,
  5) Event Notification messages used for reporting asynchronous events, 
     (including errors), and
  6) Application and Vendor Specific messages which are used to exchange 
     application data between CE and FE application endpoints, and extend the 
     protocol beyond its current capabilities. 

  Each class consists of a set of related message types. 

 audu               Expires December 23, 2004                 Page 9

Internet-Draft                   ForCES-PL               June 24 2004

  The valid message classes are:

  Message Class: 8 bits (unsigned integer)

      0      Reserved 
      1      PE  Connection and association  (CA) Messages
      2      Capabilities Control (CC) Messages 
      3      PE State Maintenance  (SM ) Message 
      4      PE Traffic Maintenance (TM) Messages 
      5      Event Notification (EN) Messages                              
      6      Vendor Specific (AV) Messages 
      7- 255 Reserved by IETF for future use

  The message types (5 bits) for the defined message classes are as follows:

  Message Type for Connection and Association (CA) Message Class 

      0      Reserved                  
      1      Join Request           
      2      Join Response
      3      Leave Request                                          
      4      Leave Response
      5      Join MCastAddr Request
      6      Join MCastAddr Response
      7      Leave MCastAddr Request
      8      Leave MCastAddr Response
     9-255   Reserved by IETF for future use

  Message Type for Capabilities Control (CC) Message Class

      0      Reserved                  
      1      Capability Request                                     
      2      Capability Response                                       
      3      Configure Request         
      4      Configure Response.                       
      5      Topology Request  
      6      Topology Response    
      7      Query Request
      8      Query Response
      9 -255 Reserved by IETF for future use

  Message Types for PE State Maintenance (SM) Message Class 
      0      Reserved                  
      1      PE Up
      2      PE Up Ack
      3      PE Down
      4      PE Down Ack
      5      PE Active                 
      6      PE Active ACK                                             

 audu               Expires December 23, 2004                 Page 10

Internet-Draft                   ForCES-PL               June 24 2004

      7      PE Inactive                          
      8      PE Inactive ACK                                              
      9      Heartbeat 
     10      Heartbeat Ack
     11-255  Reserved by IETF for future use

  Message Types for PE Traffic Maintenance (TM) Message Class 

     0      Reserved  
     1      Control Packet Redirect
     2      Control Packet Forward
     3      Control Packet Forward Response
     4-255  Reserved by IETF for future use

  Message Types for Event Notification (EN) Message Class

     0      Reserved 
     1      Event Register                           
     2      Event Register Response                  
     3      Event De-register                                         
     4      Event De-register Response                       
     5      Asynchronous FE Event Notification                              
     6-255  Reserved by IETF for future use

  Message Types for Application and Vendor Specific (AV) Message Class
   
     0     Reserved
     1     AV-Data Request
     2     AV-Data Response
     3 -255 Reserved for other vendor specific messages

  Application and Vendor specific message types interpretation is beyond the 
  scope of this protocol.   

4.1.5.  Length 

  This denotes the length of the message in octets, including the message 
  header part. 

4.1.6. Element Tag

  This is used to identity a CE or an FE element in the NE for the purpose 
  of exchanging data within the system.

 audu               Expires December 23, 2004                 Page 11

Internet-Draft                   ForCES-PL               June 24 2004

  It is made up of a SET number (identifying the group or set the source 
  element belongs to), and a unique Identifier of that element in the set. 
  Thus for a CE or FE, the Element-Tag would look like the following:

        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    a)  | CE-Set Number | CE-Identifier | 
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   b)   | FE-Set Number | FE-Identifier | 
        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Fig 2: Showing element-tags for CE and FE

  During the pre-association phase, the CEs and FEs are configured by the 
  CE-Manager and FE-Manager respectively into groups or sets. A group can 
  contain one or more elements, and it is identified by a group number 
  (CE-SET or FE-SET number). Any element within a group also has a number to 
  identify it. The combination of this identifier and the set number is the 
  element-tag number.

4.1.7. Source Tag

  This denotes the element-tag of the source of the message being exchanged 
  between a communicating CE and FE peer.

4.1.8. Destination Tag

  This denotes the element-tag of the destination or sink of the message 
  being exchanged between a communicating CE and FE peer. 

4.1.9.  Transaction Sequence Number (TSN)

  This 32-bit field uniquely identifies a transaction between an FE and a
  CE. When an endpoint makes a request it generates a TSN for use in the 
  request message; the other endpoint copies this same TSN number in its 
  reply message.  Requests using TSNs are usable by both the CE and FE.

 audu               Expires December 23, 2004                 Page 12

Internet-Draft                   ForCES-PL               June 24 2004

4.2.  Service Data or Payload Structure

  Forces-PL protocol messages consist of the Common Message Header described 
  in the previous section, followed by zero or more variable length 
  parameters, as defined by the message type. This constitutes the 
  Payload or Service Data. All Forces-PL messages are 32-bit aligned.

  Examples of the service data are the following:
    * LFB configuration 
    * LFB status and events
    * FE capability and topology information 
    * Control protocol packets

  The variable length parameters in the payload are defined in a 
  Tag-Length-Value (TLV) format as shown below:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |          Parameter Tag        |       Parameter Length        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  \                                                               \
  /                       Parameter Value                         /
  \                                                               \
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Mandatory parameters MUST be placed before optional parameters in 
  a message.

  Parameter Tag: 16 bits 

  The Tag field is a 16-bit unique identifier of the type of the parameter.  
  It takes a value of 0 to 65534. Appendix-1 lists all used Tag values and
  related messages. Values other than those defined in specific parameter 
  description are reserved for use by the IETF.

  Parameter Length: 16 bits 

  The Parameter Length field contains the size of the parameter in
  bytes, including the Parameter Tag, Parameter Length, and Parameter
  Value fields.  The Parameter Length does not include any padding bytes.

  Parameter Value: variable-length

  The Parameter Value field contains the actual information to be
  transferred in the parameter.

  The total length of a parameter (including Tag, Parameter Length and Value
  fields) MUST be a multiple of 4 bytes. If the length of the parameter is
  not a multiple of 4 bytes, the sender pads the parameter at the end (i.e., 
  after the Parameter Value field) with all zero bytes. The length of the  
  padding is NOT included in the parameter length field. A sender MUST NEVER 
  pad with more than 3 bytes. The receiver MUST ignore the padding bytes. 
 audu               Expires December 23, 2004                 Page 13

Internet-Draft                   ForCES-PL               June 24 2004

5. Forces-PL Messages

  This section defines the messages and their parameter contents.

5.1.  Association and Connection (CA) Messages

5.1.1.  Join Request 

  After the pre-association phase [section 9], the FEs can join or leave  
  any CE in a CE-set. An FE uses the Join Request message to initiate 
  association with a CE in a CE-set. The message contains the requester's 
  identity that was configured during pre-association. After a successful 
  join process, FEs can report their capabilities to the CE.   
 
  At a given point, CEs from one CE set can communicate with an FE. The FE 
  has to know which CE's requests it can accept. This information is 
  configured during the pre-association phase, during which a list of CEs 
  allowed to control the FE is configured in the FE. The FE uses this CE-list
  to send the join request. It first tries one of the CE's in the list and 
  if it not successful, it tries the next CE in the list. If all of the CEs 
  in the list are tried without success, the FE should start over again until 
  the Retry timer expires. 

  The format of the JOIN Message payload is as follows:
   
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x20)          |             Length (8)        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                            Address                            <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 The Join request Body

   Address: The Interconnect dependent unique address of the FE. 
            The tag value determines the type of addressing used.
            Valid values are:
        0x20  -- IPv4 (32 bits)
        0x30  -- IEEE (48 bits)
        0x80  -- IPV6 (128 bits)
        0xFF  -- UNKNOWN

 audu               Expires December 23, 2004                 Page 14

Internet-Draft                   ForCES-PL               June 24 2004

5.1.2.  Join Response 

  After receiving a join request message, the CE performs the following 
  operations. 
  (1) Matches the FE" "s element-tag in the request message against those 
      in its database. If not found, then the CE generates a unique 
      association ID for that FE (to denote the CE-FE connection) and 
      allocates resources for its attributes in its database. This is a 
      cold-start operation.

  (2) If the FE’s element-tag was found in the CE’s database, then the 
      CE checks up its previously stored configuration for that FE. If it 
      has any, it will use that to configure the FE in subsequent messages.   
      This is warm restart operation.                       

  (3) If the CE needs to reject the join request for some reason, it sends a 
      Leave Response Message as indicated later.

  After a successful JOIN operation, the CE should initiate the next phase
  of the association establishment process by querying the FE for its 
  capabilities, its topologies, etc, and then configuring the FE for the 
  functions it is to perform in the NE.

  The format for the JOIN RESPONSE message payload is as follows:

   0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x11)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |         FE identifier         |            FE Behavior        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                        Association ID                         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                      Heartbeat Interval                       |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                  Association Expiry Timer                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  | FE Behavior Expiry Timer (optional)                     | Prio|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Association-ID: This uniquely identifies the stream or virtual
           connection (wire) between the FE and CE. Further 
           communication between this FE-CE peer uses the stream.
                 
    FE Behavior: This defines the FE behavior when all the CEs are       
           down. A value of 1 indicates the FE should continue
           forwarding packets; a value of 0 indicates the FE
           should stop forwarding packets when the CEs are down.

 audu               Expires December 23, 2004                 Page 15

Internet-Draft                   ForCES-PL               June 24 2004

    Heartbeat Interval: This gives the time interval for the heartbeat
           messages sent from the CE to the FE in milliseconds.
          
    Association Expiry Timer: This gives the timer value in milliseconds
           after which if the FE does not receive any heartbeat
           messages from the CE it should consider the 
           association with the CE to have expired (CE DOWN).

    FE Behavior Expiry Timer: This is an optional timer value, which
           applies in case the FE behavior is to continue forwarding
           packets when CEs are down. This value indicated the time
           in seconds for which the FE should continue forwarding
           packets without associations with any CE. Values range
           from 0x0 (don't forward) to 0xffff(forward forever)
           
   Priority: This is the priority being requested for the 
           association that may result from a successful join. 

  Note that each FE can set up multiple unicast associations by making 
  multiple Join requests, each with a different priority value. This allows 
  an FE to set up an association for control information and another one for 
  user data exchange between the CE and the FE. For this version of ForCES-PL,
  the maximum limit is two (2) associations per FE.

5.1.3.  Leave Request

  The FE can leave by sending Leave request to CE. The CE's upon receiving 
  such request releases the associated resources assigned for the FE.

  The LEAVE message contains the following parameters:

    Reason
    Info String (optional)

  The format for the LEAVE Message parameters is same as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xa)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                              Reason                           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x4)           |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                         INFO String*                          <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The format and description of the optional Info String parameter is
  the same as for the PE Up message (See Section 4.3.2.1.).

 audu               Expires December 23, 2004                 Page 16

Internet-Draft                   ForCES-PL               June 24 2004
   
  The reason parameter indicates the reason the FE (or CE) is leaving the NE.
  Valid values are as follows:

    Value    Description
    0x1      Management Inhibit (Manual Removal)
    0x2      Invalid NE group   
    0x3      FE or CE not ready.(Recipient should retry 10 ms later)  
    0x4      Max number of FEs reached (for leave response only)
    0x5      Physical (Connection) Layer Fault
    0x6      Fail-over switching (refer to Section 7.3)

  The optional INFO String parameter can be any meaningful 8-bit character 
  string (up to 255 characters). This may be used for debugging purposes.

  Note that it is possible for a CE about to leave a CE-SET to also send a 
  Leave Request. 

5.1.4.  Leave Response

  When an FE is leaving the CE, the CE generates an acknowledgment to the 
  FE in the form of a Leave Response message. A CE could have generated the 
  Leave Request. In this case, the (active) CE in the CE-set will generate 
  the response sent to the leaving CE.

  (NOTE: Currently, CE-CE communication is out of scope and the mode if CE-CE 
  communication is entirely proprietary)

  If a CE needs to reject a join request from a FE for some reason, it sends 
  a Leave Response Message to the FE as well (Refer to Section 5.1.2).

  The LEAVE Response message contains the following parameters:

    Reason
    Info String (optional)

  The format of the Leave Response message is the same as in the Leave 
  Request message (See 5.1.3)

5.1.5. Join Multicast Address (JoinMCastAddr) Request

  This used by the FE or CE to join a multicast address group. Messages sent 
  from a group member will be sent to all other members in the group.

 audu               Expires December 23, 2004                 Page 17

Internet-Draft                   ForCES-PL               June 24 2004

  The format of the JOIN MCAST_ADDR Message payload is as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x20)          |             Length (8)        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                            Address                            <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x10)          |             Length (8)        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                          Mcast_Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  reserved                               | prio|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure 4 Join Mcast Address request
  
    Address: The Interconnect dependent unique address of the FE. 
            The tag value determines the type of addressing used.
            Valid values are:
        0x20  -- IPv4 (32 bits)
        0x30  -- IEEE (48 bits)
        0x80  -- IPV6 (128 bits)
        0xFF  -- UNKNOWN
  
    Mcast_Address: This is the multicast group address being joined.

    Link Priority: This is the  value of the priority for the
             link that may result from the join request. The value
             set here will be reflected in the Prio header bits 
             for data exchanges between source and sink using the 
             requested multicast address.

5.1.6.  Join Multicast Address Response 

  This is the response sent to a requester after a successful multicast 
  address Join process. The format of the body of this message is the same 
  as in the request (see 5.1.5). 

  Note this message will typically be sent by the TML (Transport layer).

  If the join request was not successful for some reason, a Leave Multicast 
  Address response (see 5.1.8) will be sent back to the requester.

5.1.7. Leave Multicast Address Request

  This message is sent by a CE or an FE endpoint to signal its intention to 
  stop receiving multicast messages sent to a particular group address.

 audu               Expires December 23, 2004                 Page 18

Internet-Draft                   ForCES-PL               June 24 2004

  The format of the LEAVE MCAST_ADDR Message payload is as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x10)          |             Length (8)        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                          Mcast_Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Reason                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure 6: Leave Multicast Address Request

     Mcast-Adress: This is the multicast address being vacated.

     Reason: Reason for leaving the group
     Valid reason values include:
        Reason      Description
        0x1     Management Inhibit

5.1.8. Leave Multicast Address Response

  This is sent to the requesting FE or CE to acknowledge a successful 
  discontinuation of the requestor's membership in a multicast address 
  group.  It can also be sent in the event of an unsuccessful 
  join-multicast-address request. Thereafter, messages sent to that multicast
  address will not be delivered to that endpoint. 

  The LEAVE MCAST-ADDR response message contains the following parameters:
    Mcast-Address: Multicast address being vacated    
    Reason:        Reason for leaving the group
    Info String (optional)

  The format for the LEAVE MCAST-ADDR Message parameters is same as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x20)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                          Mcast_Address                        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xa)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                              Reason                           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x4)           |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                         INFO String*                          <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

              Fig 7. Leave Multicast Address Response
 audu               Expires December 23, 2004                 Page 19

Internet-Draft                   ForCES-PL               June 24 2004

   Info-String: The optional INFO String parameter can be any 
         meaningful 8-bit character string (up to 255 characters). 
         This may be used for debugging purposes.

  Valid reasons for leaving an address group are as follows:
     Reason Description
    0x1 Management Inhibit
    0x2     Invalid Address Group
        0x3     Address Server Not ready (retry after 20ms)
        0x4     Max group size reached
        0x5     User-Request

  Note: This message will typically be sent by the TML layer.

5.2. Capabilities Control (CC) Messages

  This is the next phase of the JOIN process. After an FE has been 
successfully
  accepted into a CE-SET, the CE initiates the next phase of the association 
  establishment process by querying the FE for its capabilities, its topology,
  etc, and then configuring the FE for the functions it is to perform in the 
NE

5.2.1.  Capability Request

  This message is used to request the FE to report its Capabilities to
  the CE. It consists only of the header part (zero length body).

  The FE may have one or more LFBs on the forwarding plane like meter, shaper,
  egress port etc. CE may configure or query these components and their status
  at any time.  In order to do this, CE needs to know the LFBs placement and 
  sequence in the forwarding data path. The FE Model document [FE-MODEL]] 
  describes the arrangement and the relationship of those components.

  For the sake of clarity, we provide the following summary:

  An FE identifier identifies FE; The FE may contain one or more parallel 
  data path(s). On each parallel data path, there may be one or more LFBs and 
  each one is connected to another either in series or in parallel fashion. 
  The LFB is uniquely identified by a number. Examples of LFBs are filter, 
  shaper, egress port etc. 

5.2.2.  Capability Response

  This is used by the FE to report its capabilities to the CE as per CE's 
  request. This message structure might change depending on the FE Model 
  [FE-MODEL]].

 audu               Expires December 23, 2004                 Page 20

Internet-Draft                   ForCES-PL               June 24 2004

  The format for the Capability Response is as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x06)          |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      LFB Count                                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                       LFB Info                                <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     The format for the LFB Info is as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    LFB Handle                                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     LFB Type                                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  DownStream L.Comp. Count                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  DownStream L.Comp. Handle                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   .                                                               .
   |                  DownStream L.Comp. Handle                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

5.2.3.  Configure Request

  This message is used by the CE to configure the FE. The FE consists of LFBs 
  that could be configured to achieve a desired behavior of the FE in the 
  network. Some configurable attributes of the FE include LFB parameters. 
  Examples of such attributes are:
    * Port attributes (direction, bandwidth,..)
    * routing table functions
    * High Touch functions
    * Off-Load functions
    * Filter functions

  The CE might also have received a command (either through CLI or through 
  SNMP etc) to setup some tunnels or paths or some configuration through the 
  NE. This may sometimes involve configuring more than one FE. For example, 
  packets may arrive through one FE and egress the NE through another. In 
  this situation, the CE has to configure both FEs’ LFBs. If one of the 
  configuration operation fails, the CE has to perform rollback operation 
  and issue a command failure notification to the management station or CLI 
  or SNMP etc.  

 audu               Expires December 23, 2004                 Page 21

Internet-Draft                   ForCES-PL               June 24 2004

  This operation is called 2-phase commit. To perform this, CE sends series 
  of commands to each FE with command bundling bit set. Each FE after 
  receiving the command will have to save the current configuration and check
  whether it can program the requested configuration. A status message should 
  be sent back to the CE. Once CE receives all the status messages, it can 
  then send an execute command with same transaction sequence number, 
  signaling the FEs to now switch to the new configuration. 

  Command bundling refers to the ability to send an ordered set of commands 
  to the FE. Forces-PL supports command bundling via multiple TLVs in its 
  payload as described in section 4.2. Each command is formatted as a TLV 
  structure shown below, and multiple commands are sent to the FE in a 
  single Configure Request message.

  The format of the Configure Request is as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x07)          |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    C-Operation|   C-Command   |         reserved              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
   |                           LFB Handle                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
   >                            Command Data                       <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     C-Operation:       
        FEs and CEs may engage in two-phase commit operation. This
        field provides the stage of such transactions.  
            
     0x00 - Command is single operation 
     0x01 - This command is a two-phase operation, FE needs to  
            save the previous state if rollback operation may be
            performed later by FE.
     0x10 - Rollback to the previous state. 
     0x11 - Execute and complete the command.  

  During this operation the unique TSN value in the common header is     
  the same and is used to identify the transaction. Note that TSNs are 
  only unique in combination with a CE identifier, that is, two 
  separate CEs using the same TSN are considered different transactions.
    
   Configuration-command:
      This field defines the command type. Its valid values:       
       0   Reserved
       1    NULL
       2    Add
       3    Update
       4    Delete
       5    Delete All (Flush or Purge)  

 audu               Expires December 23, 2004                 Page 21

Internet-Draft                   ForCES-PL               June 24 2004

  LFB Handle:
      This field defines the LFB handle for which this command is 
      being issued.

  Command Data:
      This is the variable length configuration data for the LFB. 
      This can be encapsulated using TLV or OID or XML formats.

5.2.4.  Configure Response

  This is sent by the FE to CE to acknowledge LFB configuration request by 
CE. 
  The format of the Configure Response is as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x1d)          |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    C-Operation|   Configuration Command       |    Result     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
   |                    Logical Component Handle                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
   |           Tag (0x4)           |            Length             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                         INFO String*                          <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   The response contains the information from the command, but with
   "Result" field filled in to indicate success or failure.

   Result Value          Meaning
   CNFG-OK       0x0     Success
   CNFG-BADCMD   0x1     Bad or unsupported configuration command
   CNFG-BADPRM   0x2     Bad configuration parameter
   CNFG-NORESRC  0x3     FE Out of resources
   CNFG-FEOOS    0x4     FE Out of service

   Logical Component Handle:
      This field defines the logical component handle or identifier 
      for which this command is being issued.

   Info-String: This is an optional string used to convey more details 
      about the command result, especially in the case of failure. It 
      is character string of up to 255 characters in length. This could 
      be used for debugging purposes.
  
5.2.5.  Topology Request

  CE wants to know how each FE is connected or configured during the 
  pre-association phase. This may be used by the CE to control and configure 
  the different FEs correctly. This message consists of the Forces-PL header 
  with the Topology Request message type.
 audu               Expires December 23, 2004                 Page 23

Internet-Draft                   ForCES-PL               June 24 2004

5.2.6.  Topology Response

  This message is sent from the FE to CE in response to the Topology Request. 
  It provides the CE with the topology information of how the FEs are 
  connected to each other.
  The format for the Topology Response is as follows:

   0                   1                   2                   3 
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x1e)          |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                        Topology Info                          <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The exact content of the topology information is addressed in the 
  [FE-Model] document. A possible format shown below consists of a list of 
  FE-ID s of the FE s directly connected to the communicating FE.

  The format of the topology Info is as follows:
                                                      
   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | #of neighbor FEs                                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | FE1 ID                                                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | FE2 ID                                                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   .                                                               .
   .                                                               .
   | FEN ID                                                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

5.2.7. Query Request

  CE may be interested in querying the properties of Logical Components or 
  collecting statistical information from FE, including that of its logical 
  components. In this case, a CE sends a Query Request Message to a FE and 
  expects a Query Response Message from it. 

 audu               Expires December 23, 2004                 Page 24

Internet-Draft                   ForCES-PL               June 24 2004

  The format for the Query Request Message parameters is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x13)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |          Type of Info         |       NumOfCompoents          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x23)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Logical component Handle 1                    |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                     Query Specific Data 1                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x23)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Logical component Handle2                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                     Query Specific Data 2                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  :                               :
  |           Tag (0x23)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Logical component HandleN                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                     Query Specific Data N                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  There can be multiple logical component IDs in one message.

  Valid values of the Info Type are:
    GEN-STATS   (0x0) - General Statistics
    PORT-STATS  (0x1) - Port Statistics
    LINK-STATS  (0x2) - Link Statistics
    LComp-STATS (0x3) - Logical Component Statistics
    LComp-PROPS (0x4) - Logical Component Properties
    PORT-PROPS  (0x5) - Port Properties

  Query Specific Data: This is query specific data, which can be in 
  encapsulated as TLV or OID or XML.

5.2.8. Query Response

  After receiving a Query Request Message, a FE replies with the Query 
  Response Message. The format of the Query Response Message is as follows:

 audu               Expires December 23, 2004                 Page 25

Internet-Draft                   ForCES-PL               June 24 2004
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x14)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |            Info Type          |     NumOfComponents           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x24)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                     Logical component Handle                  |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Logical component Data                        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                                                               <
  |           Tag (0x24)          |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                     Logical component Handle                  |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Logical component Data                        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The Logical Component data is encapsulated similar to the query specific 
  data in the respective format.

5.2.9. Error TLV

  The Error TLV is used to notify the CE or FE of an error associated with 
  an incoming synchronous Request message. For example, the message might be 
  unexpected, given the current state, or a parameter value might be invalid. 
  This Error TLV can be sent as a result of a request (synchronous), or it 
  could be triggered as a result of asynchronous events occuring in the 
system 
 
  The format of the Error TLV is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xc)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                           Error Code                          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The Error Code parameter indicates the reason for the error.

  Possible error parameter values include:
    INV-PROT   (0x01) - Invalid Protocol Version        
    INV-ASSOC  (0x02) - Invalid Association ID
    BAD-MSGCLS (0x03) - Unsupported message class
    BAD-MSGTYP (0x04) - Unsupported message type
    UNEX-MSG   (0x05) - Unexpected message
    PROT-ERROR (0x06) - Protocol Error
    TWOPHASE-ERR (0x07) - Could not complete two-phase command
    COMM-FAIL   (0x08) - Communication lost between CE and FE
    BAD-TRAF-MODE (0x09) - Unsupported Traffic Mode
 audu               Expires December 23, 2004                 Page 26

Internet-Draft                   ForCES-PL               June 24 2004
  
5.3. Protocol Element (PE) Maintenance Messages

  This subsection describes the messages used to maintain the state of the 
  protocol elements (FE and CE) in the NE.

5.3.1. Protocol Element Up (PEUP)

  The UP message is sent by a PE to indicate to its master (or slave)
  that it is UP (in-service) and ready to be used.

  The format of the PEUP message is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xb)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                          INFO String*                         <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The UP message contains the following parameters
     INFO String (Optional)

  The optional Info String parameter can be any meaningful 8-bit character 
  string, up to 255 characters in length. 

5.3.2. Protocol Element Up Acknowledge (PEUP-ACK)

  The PEUP Acknowledgement message is used to acknowledge a PE-Up message 
  received from a remote PE slave or master peer.

  The PEUP Acknowledgement message contains the following parameters:

     INFO String (Optional)

  The format for the PEUP Acknowledgement message is the same as in PE UP 
  Message. 

5.3.3. Protocol Element Active (PEACT)

  The ACT message is sent by a controlling CE to ask its slave FE to go 
  ACTIVE and start handling traffic. The FE must be UP and must have sent 
  the PEUP message to the CE. This message is used to trigger the 
  establishment of the data channel between the FE and CE.

  The ACT message contains the following parameters:
 
     Traffic Mode Type 
     INFO String (Optional)

 audu               Expires December 23, 2004                 Page 27

Internet-Draft                   ForCES-PL               June 24 2004

   The format for the ACT message payload is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xb)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |      reserved                 |    Traffic Mode Type          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |         Tag (0x4)             |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                          INFO String*                         <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The Traffic Mode Type parameter identifies the failover mode of the FE 
  within an NE.  The valid values for Type are shown in the following table:

     Value     Description
      0x1      Simplex Mode (Unreplicated)
      0x2      Redundant Active-Cold-Standby 
      0x3      Redundant Active-Hot-Standby 
      0x4      Load-shared

  Within a particular NE, only one Traffic Mode Type can be used. In 
  Active-Cold-Standby,  Over-ride value indicates that the CE is over-riding 
  the current E.  

  In Simplex Mode of operation, the active element is the only one in service.
  Any removal of this element will cause service outage.

  In a Redundant Active/Cold-Standby mode, an inactive but in-service element 
  will take over from the active element when the active element goes down or
  it is removed from service. However, its state database may need to be 
  updated before being launched to active. This is the same as 
  Over-ride-weak-consistency mode of operation.
 
  In a Redundant Active-Hot-Standby mode, an inactive but in-service element 
  will take over track handling duties when the active element goes down or 
  is removed from service. It has all the necessary state information needed 
  to assume active duty instantly. This mode is recommended for minimum 
  service downtime. This is the same mode as the Over-ride-strong-consistency 
  mode.

  In Load-Shared mode, this active element is sharing traffic burden with 
  other active elements in the NE. The algorithm for distributing the traffic 
  amongst the active elements is beyond the scope of this document.

  The optional Info String parameter can be any meaningful 8-bit character 
  string, up to 255 characters in length. 

 audu               Expires December 23, 2004                 Page 28

Internet-Draft                   ForCES-PL               June 24 2004

5.3.4. Protocol Element Active Acknowledge (PEACT-ACK)

  The ACT Acknowledgement message is used to acknowledge a PE-Active message 
  received from a remote CE master. 

  The ACT Acknowledgement message contains the following parameters:
     Traffic Mode Type 
     INFO String (Optional)

  The format for the ACT Acknowledgement message is the same as in PE Active 
  Message (See 5.3.3)

5.3.5. Protocol Element Inactive (PEINACT)

  The INACT message is sent by a controlling CE to ask its slave FE to go 
  INACTIVE and stop handling traffic. After receiving this message, the FE 
  shuts down the Data channel with the CE.

  The INACT message contains the following parameters:
  
      Traffic Mode Type 
      INFO String (Optional)

  The format for the CE/FE Inactive message parameters is as shown for 
  FE/CE Active Message(See 5.3.3)

5.3.6. Protocol Element Inactive Acknowledge (PEINACT-ACK)

  The INACT Acknowledgement message is used to acknowledge a PE-Inactive 
  message received from a remote CE master. 

  The INACT-ACK  message contains the following parameters:

     Traffic Mode Type
     INFO String (Optional)

  The format for the FE or CE Inactive Acknowledgement message parameters is 
  same as  CE or FE Active Message (See 5.3.3)

5.3.7. Protocol Element Down (PEDOWN)

  Due to failure or maintenance operation, an FE can send this DOWN message 
  to its primary CE. Upon receiving this request, primary CE may reassign the 
  responsibility to other FE’s (if possible).  

  Similarly CE in a CE-set can generate the same message to all other CE's in 
  the same CE-set.  

 audu               Expires December 23, 2004                 Page 29

Internet-Draft                   ForCES-PL               June 24 2004

  The DOWN message contains the following parameters:
    Reason      - (Mandatory) reason for going down
    Info String - (Optional) information to augment reason

  The format for the DOWN Message is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xc)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                              Reason                           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x4)           |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                         INFO String*                          <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  The reason parameter indicates the reason the PE is leaving the NE.
  Valid values are as follows:

     Value        Description
     0x1          Management Inhibit (Manual Removal)
     0x2      Device Fault
     
  The format and description of the optional Info String parameter is
  the same as for the PE Up message (See Section 5.3.1).

5.3.8. Protocol Element Down Acknowledge (PEDOWN-ACK)

  The PEDN-ACK message is used by the primary CE to acknowledge the PEDN 
  message sent by the outgoing PE. The format of this message is the same 
  as for the PEDN message.

  The DOWN ACK message contains the following parameters:
    Reason      - (Mandatory) reason for going down
    Info String - (Optional) information to augment reason

  The format for the DOWN ACK Message is same as for DOWN message.

5.3.9. Heartbeat

  CE periodically polls each FE to ensure that it is operational. A CE starts
  generating these messages after the PE Active message has been sent to the 
  FE. The timers for these messages are configurable during pre-configuration
  and can be different for the active and standby CEs. The heartbeat interval 
  for a standby CE can be much larger than that of the active CE.   

  audu               Expires December 23, 2004                 Page 30

Internet-Draft                   ForCES-PL               June 24 2004

  An optional Heartbeat Data parameter may be sent in the heart beat message. 
  Its contents are defined by the sending node and are simply echoed back by 
  the receiving FE via the HB-ACK message (see below). Examples of values 
  encoded in the Heartbeat Data field by FEs could include a Heartbeat 
  Sequence Number or Timestamp.  The receiver of a Heartbeat message does not 
  process this field as it is only of significance to the sender. The receiver
  MUST respond with a Heartbeat Acknowledgement message.

  The format for the Heartbeat Message is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0x9)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                         Heartbeat  Data                       <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Note that this function can also be done by the TML. 

5.3.10. Heartbeat Acknowledge (HB-ACK)

  The receiving FE simply echo " "s the original heartbeat message back to 
  the sender.

5.4.  PE Traffic Maintenance (TM) Messages

  These messages are sent over the data channel. The data channel is 
  established after the PE Active message is sent from the CE to FE.

5.4.1. Control Packet Redirect to CE From FE

  When a Router receives both control and data packets through a physical 
  port, any of the following scenarios may occur:
 
  (a) Forwarding blade receives IP packet that is not destined for it; these 
      packets are forwarded to the CE by the forwarding plane component.
 
  (b) Forwarding blade receives IP packet that is destined for it. These 
      packets are not forwarded to the Control plane; rather they are 
      processed by the forwarding plane control logic (stack in the 
      forwarding plane). Example of such packet is ping request.

  (c) Forwarding blade receives IP packets that may be routing protocol 
      packets or packets which cannot be processed by the stack in the line 
      card. Such packets have to be forwarded to the control plane by the FE. 

  audu               Expires December 23, 2004                 Page 31

Internet-Draft                   ForCES-PL               June 24 2004

  The format of the Control Packet Redirect is as follows:

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xe)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Port ID on which packet arrived                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |           Tag (0xe)           |             Length            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  >                           Control Packet                      <
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Control Packet: The control packet that the network element
                  received through a particular IP interface.

5.4.2. Control Packet Forwarding to FE from CE

  CE may generate a packet and want FE to forward that packet through a 
  particular or multiple egress port(s). Examples of such packets are 
  routing protocol updates, discoveries, etc.

  Before generating such request, CE has to know the FE's logical components 
  and the list of available port and the configuration status. 
  (reference snapshot of Logical components )
 
   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0xf)           |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Egress Port ID through which to forward packet         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0xe)           |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                   Packet to be forwarded                      <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Packet to be forwarded is preceded by the egress port through  
  which the packet must be forwarded by the FE. 

5.4.3. Control Packet Forwarding Acknowledgement

  This is used by the FE to acknowledge the Control Packet Forwarding
  Request initiated by the CE.

  The format of this message is the same as for the Control Packet Forwarding 
  Request message.

  audu               Expires December 23, 2004                 Page 32

Internet-Draft                   ForCES-PL               June 24 2004

5.5. Event Notification Messages

  Various events in the data path can be monitored for by the FE and 
  reported to the CE. The CE must first inform the FE which of these events 
  it is interested in through a registration process. 

5.5.1. Event Register

  This is sent by the CE to the FE to request that FE notify the CE when 
  the indicated events occur on the FE. The format of the payload is as 
  follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0xe)           |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Event Type                                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                 Event Specific Data                           <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Event Type is the type of the event to report. Valid values are :
        NO-EVENTS   (0x0) -  No Events reported
    PKT-EVENTS  (0x1) -  Packet events (e.g. data errors)
    PORT-EVENTS (0x2) -  Port Events (e.g. port status changes)
        LINK-EVENTS (0x3) -  Link related events (link status changes) 
        GFE-EVENTS  (0x4) -  Generic FE events (events within FE itself)
        RSRC-USAGE  (0x5) -  System Specific levels (CPU, buffer usage)
        LC-EVENTS   (0x6) -  Logical Component events
    ALL-EVENTS  (0x8) -  All events reported

  Event Specific Data: This is the variable length event specific data, 
  which can be encapsulated using TLV or OID or XML formats. For example, 
  this could be used to specify what packets should be redirected to the CE. 
  This may also be a regular expression.

5.5.2. Event Register Acknowledgement

  This is used by the FE to acknowledge CE's event registration request. 
  The format of this payload is same as in the Event Register Request

5.5.3. Event De-Register

  This is sent by the CE to the FE to indicate that it no longer is 
  interested in receiving notifies for the events indicated in message. The 
  format of this payload is same as in the Event Register Request. After 
  receiving this message the FE SHOULD not deliver further events to the CE 
  and MUST send an Event De-Register Acknowledgement to the CE.

  audu               Expires December 23, 2004                 Page 33

Internet-Draft                   ForCES-PL               June 24 2004

5.5.4.  Event De-Register Acknowledgement

  The FE sends this message to the CE to acknowledge CE's request not get 
  any more notifies for events indicated in message. The format for this 
  payload is same as in the Event Register Request. After a FE transmits this
  message to a CE it MUST not deliver any further events until a new event 
  register is received and acknowledged.

5.5.5. Asynchronous Event Notification

  This is used to report asynchronous events occurring in the FE. These could 
  be overall FE errors, Port/Link errors or Logical Component specific events.
  The message contains the following:
    Event Type - same as that defined for Event Register
    Event ID
    Logical Component Handle 
    Diagnostic Info  - this describes the event in more detail.

  The format of the ASYNC-NTFY message is as follows: 

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x1e)          |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Event Type              |          Event ID             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 Logical Component Handle                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x7)           |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                        Diagnostic Info                        <
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Valid values of the Event ID are as follows:
    
        CPU-USAGE   (0x1) - CPU usage more than 75% of capacity
        FE-ERROR    (0x2) - FE in non-catastrophic error state
    PORT-DOWN   (0x3) - Port is down.
    PORT-UP     (0x4) - Port is up.
    PORT-ERROR  (0x5) - Port is in non-catastrophic error state
    LINK-DOWN   (0x6) - Link attached to port is down; port is up.
    LINK-UP     (0x7) - Link attached to port is up; port is up. 
        LINK-ERROR  (0x8) - Link attached to port is in error state.
        PRI-CE-DOWN (0x9) - Primary CE has gone DOWN.
        PRI-CE-DOWN (0xa) - Other LC specific.
    SECURITY-ALERT (0xb) - Possible DOS due to high resource use
     
  audu               Expires December 23, 2004                 Page 34

Internet-Draft                   ForCES-PL               June 24 2004

5.6. Application And Vendor Specific Function Message Handling

  This allows application messages to be transported opaquely over the 
  ForCES-PL protocol. Also, it allows extensions to the FE functions so that 
  new, currently unknown FE functionality (outside of those already specified)
  can be expressed. These messages will be transported transparently by the 
  ForCEs-PL protocol. Interpretation of the transported messages will be left
  solely to the application layers sitting on ForCES-PL in the CE and FE.

5.6.1. Application And Vendor Specific Data (AV-DATA Request)

  Separated PEs may use this message to pass any information that is not to 
  be consumed by ForCES-PL to each other. This message is not destined outside
  the involved PEs either. Application layers sitting on top of the FoRCES-PL 
  protocol layer can exchange information with this message between PEs.

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Tag (0x4e)          |             Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   >                   Data to be transported                      <  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  As this message is opaque to ForCES-PL, the content is vendor-specific. 
  ForCES-PL does not parse the content of this message. 

5.6.2. Vendor Specific Data Ack (VS-Data Ack)

  This is used by the PE that receives an Inter-PE communication message to 
  acknowledge the reception to the original sender.

  The format of this message is same as for VS-DATA Request.

6. Procedures for Forces-PL Protocol

6.1. CE and FE State Maintenance

  Forces-PL layer on the CE needs to maintain the states of the FEs it 
  communicates with. Likewise, the Forces-PL layer on the FE needs to maintain
  the states of the CEs the FE communicates with. The state of the (logical) 
  NE also needs to be maintained. Since the NE is comprised of CEs and FEs, 
  the NE state will be determined by the states of the contained FE and CE 
  elements.

  audu               Expires December 23, 2004                 Page 35

Internet-Draft                   ForCES-PL               June 24 2004

  Figure 2 below shows a hypothetical NE with its set of CEs and FEs

        +---------------------------------------------------------+
        | NE                                      |
        |          +--------+           +---------+               |
        |          |  CE1   |           |  CE2    |               |
        |          |(active)|           |(standby)|               |
        |          +--------+           +---------+               |
        |        ^   ^                 ^   ^                  |
        |        |   |                 |   |                  |
        |            |   +-----+  |--------+   |                  |
        |        | |-------|--+ |----------+                  | 
        |        v v       v    v                     | 
        |        +-------+    +------+  +--------+   +---------+  |
        |        | FE1   |    | FE2  |  |   FE3  |   | FE4     |  |
        |        |(act)  | -->|(act) |  |(standby)|  |(standby)|  |
        |        +-------+    +------+  +--------+   +---------+  |
        |           ^            |                    |      
        |           |            |                    |     
        +-----------|------------|--------------------------------+
                |            v

    Figure 2. Showing logical NE and its components

6.1.1. CE and FE States

  The state of each configured FE and CE is maintained by the Forces-PL layer.
  The state of each CE or FE element can change due to the following events:
 
   * Reception of management messages by CE.
   * Reception of management messages by FE.
   * Reception of control messages by FE from CE.
   * Loss of communication between CE and FE (e.g. due to faults).

  The CE and FE state transition diagram is shown in figure 3.

  audu               Expires December 23, 2004                 Page 36

Internet-Draft                   ForCES-PL               June 24 2004

                                    +-------------+
              +---------------------|             |
              |  Alternate  +-------|  PE-ACTIVE  |
              |      PE     |       +-------------+
              |   Takeover  |           ^     |
              |             |    CE/FE  |     | CE/FE
              |             |    Active |     | Inactive
              |             |           |     v
              |             |       +-------------+
              |             |       |             |
              |             +------>|   PE-INACT  |
              |                     +-------------+
              |                         ^    |
  CE/FE Down/ |                  CE/FE  |    | CE/FE Down /
   CE-FE COMM |                    Up   |    | CE-FE COMM FAIL
    FAIL      |                         |    v
              |                     +-------------+
              +-------------------->|             |
                                    |   PE-DOWN   |
                                    +-------------+

    Figure 3. CE and FE State Transition Diagram

  The possible states of a protocol element (CE or FE) are:

   PE-DOWN: CE or FE is unavailable for service and/or the related CE-FE
       association is down. Initially, all CEs and FEs will be in this state. 
       A CE or FE in this state should not be sent any traffic messages.

   PE-INACTIVE: The CE or FE is available for service and the related
       CE-FE ForCES-PL association is up, but application traffic is stopped 
       (the CE or FE could be in a standby state for example). In this state, 
       the CE or FE involved can be sent management, control, and non-traffic
       related messages.

   PE-ACTIVE: The CE or FE is available, and actively carrying application 
       traffic.

6.2. State Maintenance Procedures

  Before the establishment of a CE-FE association, the CE must be in-service 
  and active but the FE is Down. Local management (CE Manager or FE Manager) 
  can be used to effect appropriate state transitions of CEs and FEs.

6.2.1. Protocol Element Up

  After an FE has successfully established an association with a CE, the FE 
  sends a PE-UP message to indicate to the CE that it has finished all its 
  internal configuration and is available.

  audu               Expires December 23, 2004                 Page 37

Internet-Draft                   ForCES-PL               June 24 2004
  When the CE gets the PE-UP message, and the FE is not locked out for local 
  management reasons, Forces-PL at the CE will mark the FE as UP but 
  "Inactive". The CE responds with a PE-UP Ack message in acknowledgement. 
  If for any reason the CE cannot respond with a PE-UP, it will respond with 
  a PE-DOWN Ack message with an appropriate reason parameter.

  The CE can also generate the PE-UP message. The last ACTIVE CE may have 
  gone DOWN after establishing an association with a FE. In this case, the NE 
  would first transition into the PENDING state for a duration of T(r), and 
  then to DOWN state. The first CE that transitions to UP state will send a 
  PE-UP to the FE to notify it of its status, assuming the link between them 
  is up. The FE will acknowledge with a PE-UP Ack.

  If the source PE does not receive a response from the target PE, or if a 
  PE-DOWN Ack is received, source PE MAY resend PE-UP message until it 
  receives a PE-UP Ack from the target. The default behavior is NO 
  retransmission.

6.2.2. Protocol Element Down

  The FE will send a PE-DOWN to the CE when the FE is to be removed from the 
  list of FEs in an NE that it is a member of, that is eligible to receive 
  application traffic or management messages.

  Forces-PL at the CE marks the FE as "Down" and returns a PE-DOWN Ack message
  to the FE if one of the following events occur:

   - a PE-DOWN message is received from the FE
   - another state message is received from an FE but the FE is locked out 
     by management for some reason.

  The CE sends a PE-DOWN Ack message in response this message. If the FE does 
  not receive a response from the CE, the FE MAY send PE-DOWN messages until 
  it receives a PE-DOWN Ack message from the CE or the association goes down. 
  The default behavior is NO retransmission.

  The CE may also send a PE-DOWN messaged to the FE. This occurs when the CE 
  is about to be removed from service, and it is the ACTIVE CE. On getting 
  this notification, the FE will respond with a PE-DOWN Ack, and stop sending 
  any more messages to the out-going CE. 
  The whole mechanism allows for a graceful removal of CEs or FEs.

6.2.3. Protocol Element ACTIVE

  Any time after the CE has sent a PE-UP Ack to the FE, the CE can send a 
  PE-Active (PEACT) to the FE, to activate the FE to start processing traffic.

  When a PEACT message is received, the FE responds with a PEACT Ack message, 
  after which it starts handling traffic messages. The FE establishes the Data
  channel with the CE after this message. The FE must wait for the PEACT 
  message from the CE before handling traffic data. The CE only sends the 
  PEACT message if it intends to transition the FE to ACTIVE state.
  audu               Expires December 23, 2004                 Page 38

Internet-Draft                   ForCES-PL               June 24 2004
 
6.2.4.  Protocol Element Inactive

  Any time after the CE has sent a PE-Active to the FE, the CE can send a 
  PE-Inactive (PEINACT) to the FE, to command the FE to stop processing 
  traffic.

  When a PEINACT message is received, the FE responds with a PEINACT Ack 
  message, after which it stops handling traffic messages. The FE shuts down 
  the Data channel with the CE after it receives this message. The FE must 
  wait for another PEACT message from the CE before starting handling traffic 
  again. The CE only sends the PEINACT message if it intends to transition 
  the FE to INACTIVE state.

7. Example Scenarios

7.1. Establishment of Association 

  The associations among CEs and FEs are initiated via Join request and 
  response messages. If a join request is granted by the CE, a join response 
  message is replied to the request message. If a join request is denied, a 
  Leave response message is replied to the join request message. If CEs and 
  FEs are operating in an insecure environment then a security association 
  has to be established between them before any ForCES-PL messages can be 
  exchanged (see section 8).

  This is followed by capability query, topology query. When the FE is ready 
  to start forwarding data traffic, it sends a PE-UP message to the CE. The 
  CE responds with PE-UP Ack.  According to the configuration, the CE sends a 
  PE-ACTIVE to inform the FE to go active and start forwarding data traffic. 
  The FE acknowledges it with a PE-ACTIVE Ack and starts forwarding traffic . 
  At this point the association establishment is complete. The FE establishes 
  the data channel with the CE after this. The sequences of messages are 
  illustrated in the Figure below.

                     FE                      CE
                     |                       |                      
                     |   Join REQ            | 
                   1 |---------------------->| 
                     |                       | 
                     |   Join RESP           |
                   2 |<----------------------| 
                     |                       | 
                     | CAPABILITY Request    | 
                   3 |<----------------------| 
                     |                       | 
                     | CAPABILITY Response   | 
                   4 |---------------------->| 
                     |                       | 
                     |  TOPOLOGY Request     | 
                   5 |<----------------------| 
                     |                       | 
  audu               Expires December 23, 2004                 Page 39

Internet-Draft                   ForCES-PL               June 24 2004

                     | TOPOLOGY Response     | 
                   6 |---------------------->| 
                     |                       | 
                     |     PE UP             |   
                   7 |---------------------->| 
                     |                       | 
                     |    PE UP ACK          | 
                   8 |<----------------------| 
                     |                       | 
                     |    PE ACT             | 
                    9|<----------------------| 
                     |                       |
                     |    PE-ACT ACK         |
                   10|---------------------->| 
                     |                       |
                     |    Data channel Estb  |
                   11|<--------------------->|

        Figure 4: Association Establishment messages between CE and FE

7.2. Steady State Communication

  Once the CE and FEs establish their association and exchange initial 
  configuration information, they enter a phase of steady state communication,
  with the following example messages exchanging.
         
                    FE                      CE 
                     |                       | 
                     |Heart Beat             | 
                   1 |<--------------------->| 
                     |                       | 
                     |Heart Beat ACK         |
                   2 |<--------------------->| 
                     |                       | 
                     |  Query Request        | 
                   3 |<----------------------| 
                     |                       | 
                     |  Query Response       | 
                   4 |---------------------->| 
                     |                       | 
                     | Aysnc Event Notice    | 
                   5 |---------------------->| 
                     |                       | 
                     |Configure Request      |
                   6 |<----------------------| 
                     |                       | 
                     |Configure Response     |
                   7 |---------------------->| 
                     |                       | 
                     |Control Packet Redirect|(over data channel) 
                   8 |---------------------->| 
                                            
        Figure 5: Steady State communication between CE and FE
  audu               Expires December 23, 2004                 Page 40

Internet-Draft                   ForCES-PL               June 24 2004

  When transferring forwarding information to FE, the CE uses the configure 
  request message with the  length field indicating the number of bytes of 
  configuration information. 

7.3. CE Fail-over Scenarios

  As mentioned before, there are two basic modes of high-availability or 
  Failover support provided by Forces-PL protocol. This is configured during 
  the pre-association phase [as described in section 9]. For both cases, the 
  FE must establish association using ForCES-PL protocol [as described in 
  section 7.1] with both the primary and standby CEs in the CE set. This 
  association establishment includes the security associations described in 
  section 8, also capability and topology discovery. This helps with fast 
  failover since the FE avoids re-establishment of CE-FE association during 
  failover.

  For strong consistency, the FE establishes the control and data channels 
  with both CEs and forwards all asynchronous events and protocol control 
  packets such as RIP, OSPF packets to both CEs. But only the primary CE 
  configures and controls the FE, the standby CE uses the information provided
  by the FE to keep its state synchronized with the primary CE. When FE 
  detects failure of primary CE it informs the standby CE using the 
  asynchronous Event message. The standby CE can also obtain that information
  using some CE-to-CE protocol. In case of failure of the primary CE, the 
  standby CE takes over the control of the FE. Note that in case of strong 
  consistency, CE-to-CE protocol is not needed to keep the state in the 
  primary and standby CEs synchronized.

                     FE                      CE Primary         CE Standby
                     |                       |                    |
                     | Asso Estb(Caps, topo) |                    |
                   1 |<--------------------->|                    |
                     |                       |                    |
                     |         Asso Estb(Caps, topo exchange)     |
                   2 |<----------------------|------------------->| 
                     |                       |                    |
                     |   data + control      |                    |
                   3 |<--------------------->|                    |
                     |                       |                    |
                     |         data + control|(HeartBeats only)   |
                   4 |-----------------------|------------------->| 
                     |                       |                    |
                     |                   FAILURE                  |
                     |                                            |
                     |               PRI-CE-DOWN                  |
                   5 |------------------------------------------->| 
                     |                                            |
                     |             data + control                 |
                   6 |------------------------------------------->| 

        Figure 6: CE Failover for strong consistency mode

  audu               Expires December 23, 2004                 Page 41

Internet-Draft                   ForCES-PL               June 24 2004

  For weak consistency, the FE establishes the control channel with both CEs 
  but the data channel with only the primary CE. The only communication with 
  the standby CE is the heartbeat exchange. The standby CE keeps it state 
  synchronized using some CE-to-CE protocol. When FE detects failure of 
  primary CE it informs the standby CE using the asynchronous Event message. 
  In case of failure of the primary CE, the FE establishes the data channel 
  with the standby CE which then takes over the control of the FE. 

  Note that in both failover modes the FE does not need to store any state 
  in order to synchronize the CEs.

                     FE                      CE Primary        CE Standby
                     |                       |                    |
                     |  Asso Estb(Caps, topo)|                    |
                   1 |<--------------------->|                    |
                     |                       |                    |
                     |         Asso Estb(Caps, topo exchange)     |
                   2 |<----------------------|------------------->| 
                     |                       |                    |
                     |   data + control      |                    |
                   3 |<--------------------->|                    |
                     |                       |                    |
                     |           control|(HeartBeats only)        |
                   4 |-----------------------|------------------->| 
                     |                       |                    |
                     |                   FAILURE                  |
                     |                                            |
                     |               PRI-CE-DOWN                  |
                   5 |------------------------------------------->| 
                     |                                            |
                     |             data + control                 |
                   6 |------------------------------------------->| 

        Figure 7: CE Failover for weak consistency mode

8. Security Considerations 

  If the CE or FE are in a single box and network operator is running under a 
  secured environment then it is up to the network administrator to turn off 
  all the security functions. This is configured during the pre-association 
  phase of the protocol. 

  Whether FE and CE are in a single box or multiple-hop, rate limiter 
  mechanism should be in place to defend against the CPU bound and bandwidth 
  (network) bound DoS attacks. We recommend this for all security mechanisms. 

  When the CEs, FEs or ForCES-PL endpoints are running over IP networks or in 
  an insecure environment, ForCES-PL MUST use TLS [TLS] to provide security. 
  The security association between the CEs and FEs MUST be established before
  any ForCES-PL association establishment messages are exchanged between the 
  CEs and FEs. 

  audu               Expires December 23, 2004                 Page 42

Internet-Draft                   ForCES-PL               June 24 2004

8.1. TLS Usage with Forces-PL

  This section is applicable for CE or FE endpoints that use Forces-PL with 
  TLS [TLS-SCTP][TLS] to secure the communication.

  Since CE is master and FEs are slaves, the FEs are TLS clients and CEs are 
  TLS server. Forces-PL endpoints that implement TLS MUST perform mutual 
  authentication during TLS session establishment process. CE must request 
  certificate from FE and FE needs to pass the requested information. 

  We recommend "TLS-RSA-with-AES-128-CBC-SHA" cipher suite, but CE or FE may 
  negotiate other TLS cipher suites. TLS must be used for all control channel 
  messages. TLS is optional for the data channel since data channel packets 
  are not encrypted externally to the NE. 

  Forces-PL uses TLS to provide security when the NE is in an insecure 
  environment. This is because IPsec provides less flexibility when 
  configuring trust anchors since it is transparent to the application and 
  use of Port identifiers is prohibited within IKE Phase 1. This provides 
  restriction for IPsec to configure trust anchors for each application 
  separately and policy configuration is common for all applications.

9.  Architecture support for Forces-PL protocol 

  Pre-association phase is used to configure certain key attributes. 
  FE-Manager and CE-Manager are responsible for providing that information.  

9.1. Configurable parameters

  The following are the currently identified configurable parameters that can 
  be done through FE-Manager and CE-Manager for FE and CE's respectively

   (1) Fail over configuration (Strong, Weak consistency)
   (2) Number of CEs with which it has to communicate 
   (3) Maximum number of CE
   (4) Timer for health check
   (5) Maximum numbers of FEs that each CE can support in a NE

10. IANA Considerations
  
  ForCES-PL protocol needs to have a two well-defined port numbers, which 
  needs to be assigned by IANA.

11. References
11.1. Normative References

[STD] S. Bradner, "The Internet Standards Process-Revision 3", RFC 2026, 
      October 1996. 

  audu               Expires December 23, 2004                 Page 43

Internet-Draft                   ForCES-PL               June 24 2004

[KEYWORDS]S. Bradner, "Keywords for use in RFCs to Indicate 
      Requirement Levels", RFC2119 (BCP), IETF, March 1997.
 
[FORCES-REQ] Khosravi, et. al., "Requirements for Separation of IP 
      Control and Forwarding", rfc 3654, November 2003, rfc 3654.

[FRAMEWK] L. Yang, et. al, " ForCES Architectural Framework", 
      rfc 3746 April 2004,

[FE-MODEL] L. Yang, et. al, "ForCES Forwarding Element Functional 
      Model", work in progress, February 2004,<draft-ietf-forces-
      model-02.txt>

11.2. Informative References

[IUA] Morneault, K,. Rengasami, S,. Kalla, M., and G. Sidebottom, 
      "ISDN Q.921-User Adaptation Layer", RFC 3057, February 2001

[OSPF] J. Moy, "OSPF Version 2", RFC 2328, April 1998

[TLS] Dierks, T., Allen, C., Treese, W., Karlton, P., Freier, A. and 
      P. Kocher, "The TLS Protocol Version 1.0", RFC 2246, January 1999.

[TLS-SCTP] Jungmaier, A., Rescorla, E. and M. Tuexen, "Transport   
      Layer Security over Stream Control Transmission Protocol", 
      RFC 3436, December 2002.

[SCTP] R. Stewart, et al, "Stream Control Transmission Protocol  
     (SCTP)", RFC 2960, October 2000. 

[DCCP] E. Kohler, M. Handley, S. Floyd, J. Padhye, "Datagram
       Congestion Control Protocol (DCCP)", draft-ietf-dccp-spec-
       04.txt, June 2003.

[CONG-CONTRL] Floyd, S., "Congestion Control Principles", RFC 2914,
       September 2000.

[FACT] Audu A., Gopal R., Khosravi H., Wu C. "Forwarding and Control
       Element protocol (FACT)" draft-gopal-forces-fact-06.txt,
       February 2004.

12. Acknowledgments

  This work is based mostly on the ideas expressed in [FACT]. We hereby  
  acknowledge all the contributors to FACT, including Ram Gopal, Chaoping Wu 
  and Hormuzd Khosravi.
 
  audu               Expires December 23, 2004                 Page 44

Internet-Draft                   ForCES-PL               June 24 2004

13. Authors' Addresses

  Alex Audu
  Alcatel R&I
  3400 West Plano Parkwy
  Plano, TX 75075  
  Phone: 1-972-477-7809
  Email: [email-masked] 

Appendix-1: Tag (Hex) Values Used in Forces-PL Messages

  +-----+--------------------+---------------------------------------+
  |Tag  | Meaning            | Messages                              |
  +-----+--------------------+---------------------------------------+
  |0001 |IPv4 Join Address   | Join Request                          |
  +-----+--------------------+---------------------------------------+
  |0002 |IPv6 Join Address   | Join Request                          |
  +-----+--------------------+---------------------------------------+
  |0003 |Join Configuration  | Join Response                         |
  +-----+--------------------+---------------------------------------+
  |0004 |Leave Reason        | Leave Request/Response                |
  +-----+--------------------+---------------------------------------+
  |0005 |Info String         | Leave Req/Resp                        |
  +-----+--------------------+---------------------------------------+
  |0006 |FE Capability       | Capability Response                   |
  +-----+--------------------+---------------------------------------+
  |0007 |Config Compo Command| Configure Request                     |
  +-----+--------------------+---------------------------------------+
  |0008 |Config Compo Result | Configure Response                    |
  +-----+--------------------+---------------------------------------+
  |0009 |Topology Data       | Topology Response                     |
  +-----+--------------------+---------------------------------------+
  |000A |Query Data          | Query Request                         |
  +-----+--------------------+---------------------------------------+
  |000B |LFB Data            | Query Response                        |
  +-----+--------------------+---------------------------------------+
  |000C |Traffic Mode        |    PE (IN)ACT/ACK                     |
  +-----+--------------------+---------------------------------------+
  |000D |Data Channel Info   |    PE ACT                             |
  +-----+--------------------+---------------------------------------+
  |000E |Down Reason         | PE DOWN/ACK                           |
  +-----+--------------------+---------------------------------------+
  |000F |Heartbeat Data      | Heartbeat/Heartbeat ACK               |
  +-----+--------------------+---------------------------------------+
  |0010 |LFB ID              | CP Redirect, CP Forwarding/Resp       |
  +-----+--------------------+---------------------------------------+
  |0011 |Control Packet      | CP Redirect, CP Forwarding/Resp       |
  +-----+--------------------+---------------------------------------+
  |0012 |Log. Comp ID list   | CP Forwarding/Resp                    |
  +-----+--------------------+---------------------------------------+
  |0013 |Event Data          | Event (De-)Register Req               |
  +-----+--------------------+---------------------------------------+
  audu               Expires December 23, 2004                 Page 45

Internet-Draft                   ForCES-PL               June 24 2004

  |0014 | Diagnostic Info    | Asynchronous EE Event Notification    |
  +-----+--------------------+---------------------------------------+
  |0015 |Event-Handle ID     | Asynchronous EE Event Notification    |
  +-----+--------------------+---------------------------------------+
  |0016 |Error Code          | Join, leave, Cap, Topo Response msgs  |
  +-----+--------------------+---------------------------------------+
  |0017 |Vendor Specific Data| VS-DATA Request/Resp                  |
  +-----+--------------------+---------------------------------------+
  |0018 |Result              | Event (De-)Register Resp              |
  +-----+--------------------+---------------------------------------+

Appendix-2: Interfaces To Local Management(CE/FE Manager)

  As part of any normal protocol operation, management interface either CLI 
  or EMS or NMS or other suitable entity would like to send commands to either
  FE or CE. This section identifies minimal command set that may be required 
  for such  operation. This may be implemented by  vendors in various ways and
  is beyond the scope of ForCES protocol. This section describes the minimal 
  message and how FE and CE may use ForCES-PL to inform the local management 
  operation of each other.

M-PE-STATUS
  The status of CEs and FEs are stored in and tracked by ForCES-PL. This 
  primitive is used to request, confirm, and indicate the status of a 
  PE (FE or CE) to local management.

M-ASSOCIATION-STATUS
  This is used to request and indicate the status of the association 
  between a CE and an FE.

M-ERROR
  This is used to indicate an error with a received ForCES-PL message to 
  FE or CE Manager.

M-PE-UP
  This primitive can be used by FE or CE Manager to request that a FE or CE
  be restored into in-service (UP) state by ForCES-PL. This could be triggered
  by a RESTORE request from the CLI (Command Line Interface) into FE or CE 
  Manager.

    (CLI)----Restore(COLD)--->( FE or CE Manager)---- M-PE-UP(COLD)--->(FPL)

  Forces-PL can also use this primitive to indicate or acknowledge to FE or CE
  Managerthat a FE or CE is UP (with a M-PE-UP.indicate and M-PE-UP.confirm 
  Primitives respectively). Valid parameters to M-PE-UP.req are:
    COLD   -  initialize all attributes of PE during restore process.
    WARM   -  use previous attributes in memory to restore PE (assumes 
                  those attributes have not been lost).
    CONFIG -  Restore PE with new configuration attributes.
   
  audu               Expires December 23, 2004                 Page 46

Internet-Draft                   ForCES-PL               June 24 2004

M-PE-DOWN  
  This can be used by FE or CE Manager to request that a PE be taken to the 
  DOWN State. It could be triggered for example, by CLI sending a Remove 
  request to the local management layer.

   (CLI)---Remove(BOOT)--->(FE or CE Manager)----- M-PE-DOWN(BOOT) --->(FPL)
    
   Forces-PL can in turn indicate a PE-DOWN event or confirm the DOWN request 
   with a M-PE-DOWN.ind or M-PE-DOWN.con respectively. Valid parameters to 
   M-PE-DOWN.req are: 
    NO-BOOT - previous (static) contents in memory are preserved.
        BOOT    - all previous contents in memory are zero'd out.
        CONFIG  - previous configuration information between FE and CE are 
                  lost and new ones are being established.

M-PE-ACTIVE
  This is used by ForCES-PL to inform FE or CE Manager that FE or CE has 
  gone ACTIVE.

M-PE-INACTIVE
  This is used by ForCES-PL to inform FE or CE Manager that FE or CE has gone
  INACTIVE.

Appendix-3: NE States

  It may be necessary to track the state of the NE itself. Since the NE
  consists of distributed CEs and FEs, the state of the NE will be dependent
  on the states of its CEs and FEs. The state of the NE is maintained by 
  Forces-PL in both FE and CE. 

  The state of an NE can be changed due to events including:
    * CE or FE state transitions
    * Recovery timer triggers

  The possible states of a NE are as follows:

   NE-DOWN: The network element is not available for service. This implies 
            all related CEs and FEs are in the PE-DOWN state. Initially, 
            the NE will be in this state.

   NE-INACTIVE: The network element is available but no application traffic 
            is active. Here, one or more protocol elements (CE or FE) are in 
            the PE-INACTIVE state, but none in the PE-ACTIVE state. Also, 
            the recovery timer is not running, or has expired. This may be 
            the state of standby NE if redundancy is provided at logical NE 
            level.

   NE-ACTIVE: The network element is available and it is carrying application 
            traffic. This implies that at least, one CE-FE communicating pair 
            is in PE-ACTIVE state.

  audu               Expires December 23, 2004                 Page 47

Internet-Draft                   ForCES-PL               June 24 2004

   NE-PENDING: An active CE or FE has transitioned into inactive or down 
            state, and it was the last remaining active CE or FE in the NE.
            A recovery timer T(r), will be started, and the source FE or CE 
            will queue up messages meant for the inactive target. If another 
            target CE or FE becomes active (depending on which went inactive),
            before T(r) expires, the queued up messages are directed to the 
            newly active CE or FE, and T(r) timer is cancelled. In this case, 
            NE will move back to the NE-ACTIVE state. However, if T(r) expires
            before an alternate CE or FE becomes active, the queued up 
            messages are discarded, and the NE will move to NE-DOWN state.

        +----------+  one PE goes ACTIVE     +-------------+
        |          |------------------------>|             |
        | NE-INACT |                         |  NE-ACTIVE  |
        |          |                         |             |
        |          |<                        |             |
        +----------+ \                       +-------------+
           ^   |      \ Tr fires;                 ^    |
           |   |       \ at least one             |    |
           |   |        \ PE is UP                |    |
           |   |         \                        |    |
           |   |          \                       |    |
           |   |           \                      |    |
   one PE  |   |            \            one PE   |    | Last ACTIVE PE
   goes    |   | all PEs     \------\    goes to  |    | goes INACT
   to      |   | go DOWN             \   ACTIVE   |    | or DOWN
   INACT   |   |                      \           |    | (start Tr timer)
           |   |                       \          |    |
           |   |                        \         |    |
           |   |                         \        |    |
           |   v                          \       |    v
        +----------+                       \ +-------------+
        |          |                        -|             |
        | NE-DOWN  |                         | NE-PENDING  |
        |          |                         |  (queueing) |
        |          |