Subject: Evaluation and comments on draft-doria-forces-protocol-01.txt

From: Zsolt Haraszti <[email-masked]>
Date: Mon, 2 Aug 2004 05:12:11 -0400

As per David's request, here are my  comments  to  the  latest  ForCES
protocol draft 'draft-doria-forces-protocol-01.txt.

The current draft shows good progress. Although there  are  many  open
issues, I think the group is on the right track, and  as  such  I  can
recommend making this draft a WG draft.

What follows is my list of comments and questions.

I recognize that the the current revision is work in progress, hence I
will omit the editorial comments. I will also  disregard  the  current
inconsistencies (within the draft and between the draft and the  model
draft). I assume that these will disappear  as  the  draft  matures. I
would be delighted to review the draft at some later stages from these
perspectives.

Overall comments/questions:
---------------------------

1. I have a slight confusion regarding the scope of the  protocol.  On
   the  one  hand,  I  assume  it  is  meant  strictly  for  CE<--->FE
   communication, that  is,  covering  the  Fp  reference  point  (and
   nothing more). Some details in the draft seem to suggest otherwise.
   For example, the "all FEs and CEs" broadcasting mentioned on  pp.21
   and elsewhere. Please clarify.

2. The draft would benefit form a more detailed definition of the  TML
   and assumptions on how the PL layer will rely and interact with the
   TML. In particular the following should be covered:

a. How is TML defined? Is it the glue layer between the PL and a given
   lower-layer transport protocol, or is it defined as the lower-layer
   transport protocol itself (with an optional glue layer if  needed)?
   The definition on pp.8 suggests the former, while Section 4 implies
   the latter. Please clarify. Is it actually expected that  TML  will
   need a glue layer (header), depending on the nature and  functional
   richness of the underlying protocol?

   For clarity, from here on I will use TML*  when  referring  to  TML
   which includes the transport protocol, and TML as to refer only  to
   a glue layer.

b. Is it assumed that the TML* is end-to-end in the sense that  it  is
   terminated in the same software stack  where  the  ForCES  Protocol
   (FP) is terminated? In other words: can we assume that TML is never
   popped en route between the CE and FE?  This  has  implications  on
   what has to be in the PL and TML headers.

c. How the PL is layered on top of the TML? Certain information  needs
   to be passed down that is not  currently  in  the  PL  header  (for
   instance, the priority selector). Is the intended model  such  that
   the ForCES Protocol (FP) machine constructs both  the  PL  and  TML
   headers as one unit, then  passes  them  down  as  payload  to  the
   transport protocol (in which case I expect the priority info  being
   put in the TML header)? What will be passed down  as  part  of  the
   PL(+TML) headers, and what will be passed  down  as  function  call
   parameters?

d. TML* point-to-point addressing: Is it assumed that  the  TML*  will
   have point-to-point (unicast)  addressing  capabilities?  (I  guess
   so.) If so, how will TML* end-point addresses map to CE or FE  IDs,
   1:1 or 1:M? In other words, is it expected/required that  the  TML*
   is capable of multiplexing  multiple  concurrent  sessions  between
   more than one CE-FE pairs using the same TML* end-point  addresses?
   (Again I guess so, otherwise the CE ID or FE ID in the PL would  be
   redundant with the  TML*  endpoint  addresses,  but  it  should  be
   specified.)

   On  the  other  hand,  if  we  cannot  assume  that  the  TML*  has
   point-to-point addressing capability and 1:1  mapping  between  its
   addresses and the CE/FE IDs, then the "give me my FE ID"  mechanism
   described in 6.1.1. is broken. (How will the TML* layer know  where
   to deliver the association setup response?)

3. The draft seems to be  in  an  interim  status  regarding  how  the
   FE-level (coarse) functions and the protocol parameters  should  be
   handled.  Regarding   them   as   special   LFBs   (one   for   the
   protocol-specific info and one for the FE-specific  info)  seems  a
   good idea. The ForCES WG (and not just the  protocol  design  team)
   should make a decision on the issue ASAP. The decision has profound
   implications on the protocol, needless to say.

   Suppose the decision is  to  model  the  protocol  entity  and  the
   FE-level attributes as two special LFB type/instance  (as  proposed
   in the draft). In that case some interim  TLVs  can  be  eliminated
   (e.g., the ones defined in Figure 11).

   Moreover, on similar (purification) grounds, many of  the  protocol
   operations can be defined as attribute settings of LFBs (either the
   real LFBs or one of the two special LFBs).

   For example, registering for asynchronous event notification can be
   regarded as setting some attributes of the given  LFB  or  the  "FE
   LFB."

   Or, initializing the Heart-Beat-Interval can be done via setting  a
   "HeartBeatInterval" attribute of the "ForCES-Protocol LFB."

   Or, the  state  maintenance  messages  of  6.6  can  be  mapped  to
   setting/querying the respective attribute(s) of the "FE LFB."

4. LFB addressing: The current method is a 16bit LFB  type  ID  and  a
   16bit instance ID. (BTW, the text should define if  the  uniqueness
   scope of the instance ID here is per type or per FE.) Obviously the
   other alternative would be a flat 32-bit LFB instance  space.  What
   is the motivation of carrying the type id in  each  message  (other
   than it is helpful when debugging the protocol with a snooper)? The
   16bit instance ID has a risk to be grown out. A 16-bit type ID,  on
   the other hand, seems a bit too generous.

5. Request/response correlation and  CE-side  addressing:  This  issue
   needs to be addressed in  more  detail.  Furthermore,  the  current
   header fields seem insufficient. When an entity inside the CE sends
   a request to one (or more) FE(s), the response needs to  be  routed
   back to the requester entity inside  the  CE  (because  many  other
   requests  made  by  the  CE  may  be  pending  simultaneously).  In
   addition, the requester entity must be able to  tell  which  former
   request this response belongs to (because it may have  issued  many
   requests). The only field in the PL header that  can  be  used  for
   this purpose is the sequence number. This seems insufficient.

   Using the sequence number for this purpose can be a  major  problem
   if we want to allow response fragmentation.  Consider  the  example
   when the CE queries an FE (or many FEs) for  the  full  content  of
   some (large) counter table(s), and where the response is  over  the
   MTU of the TML*. Even if the TML allowed infinitely  long  messages
   (BTW, the draft should address what expectation the PL will have on
   TML  with  regards  to  message  size  limitations,   segmentation,
   reassembly, etc.), for  other  reasons  it  may  be  beneficial  to
   fragment the response into multiple FP messages. In this  case  the
   sequence number can no longer be used as a correlator.

   The authors may want consider using a smaller sequence  number  and
   adding a correlator field (and  optionally  a  CE  entity/subsystem
   ID).

6. The draft does not describe how LFB config and  query  multicasting
   will be implemented. Since I have not seen a  method  proposed  for
   this yet, here is  a  simple,  yet  very  efficient  mechanism  for
   consideration:

   The FE will have an "LFB alias table" which can be configured.  For
   example, this may be an attribute of the "FE LFB" or "FP LFB." Each
   entry in this table defines a virtual LFB ID and a list  of  (i.e.,
   one or more) actual (real) LFB ID(s). If a protocol request arrives
   with a virtual LFB ID, the ID is looked up in the alias  table  and
   the request is forwarded to the LFB(s) associated with the alias.

   The CE is responsible to manipulate the alias table of the FE.

   This mechanism can be used for multicasting  the  same  message  to
   multiple LFB instances (of the same type or even  different  types,
   although the latter may not be  as  practically  important).  [Note
   that when using the currently defined schema for LFB identification
   (16bit LFB type + 16bit LFB instance), one does not need to use the
   above alias table for the special case of multicasting to *all* LFB
   instances of a given type; provided that  we  reserve  a  wild-card
   instance  ID  for  this  purpose.  I  would  not  even  call   this
   multicasting, but rather "restricted broadcasting.]

   This mechanism can also be useful  to  re-map  a  particular  LFB's
   instance ID to a CE-controlled value. This can be useful if the  CE
   wants to set up a multicast tree for some LFBs across multiple FEs.
   The leaf LFBs of this tree  may  have  different  instance  IDs  by
   nature of the way instance IDs are defined by the  FEs.  Using  the
   alias table the CE can set up a virtual instance ID across the  FEs
   that map the same ID to the actual LFB ID on each FEs.

   If this is not clear, I am happy to describe this in more detail.

Other specific comments:
------------------------

pp.6., FE Model definition:  Add ", and it defines what is
   configurable in the FE."

pp.12., last para of 3.1.3:  Is this an example of point d?

pp.16., Section 3.3.2: For other potential FE attributes please take
   a look at [FE-MODEL].  The LFB graph is probably the most important
   FE attribute.

pp.19., "PL ID": Do you mean FE and CE IDs?

pp.20., "Version(4 bits)": Why to split it to two subfields?  Can you
   motivate?

pp.20., "Command (8 bits)": This is called "Message Type" in the
   Figure and elsewhere in the draft.

pp.21., line 2: What us an NE broadcasting? Example?

pp.21., lines 8-9: What do you mean by the "or a mix of both?"

pp.21., FE and CE ID address space description: Designating the FE/CE
   by a separate bit seems useful (especially considering
   snooping-based debugging).  Why do you restrict this useful
   information to unicast messages.  It seems equally useful also for
   multicast and broadcast.

pp.21.: What's the meaning of an "all FEs and CEs" broadcast?

pp.21., beginning of last para: If a message is only to some LFBs (of
   class X), then it is not a broadcast message by definition, but a
   multicast. The 1st sentence of the paragraph should start "A
   multicast message..."

pp.22., "ACK indicator (2 bits)": better be called a "SEND-ACK" or
   "ACK-REQ".

pp.23., Section 6.1.1: Please motivate why the association setup is
   initiated by the FE (rather than the CE).  The FP being a
   master-slave protocol where the CE is the master, this choice
   here seems "against the flow."

pp.23., optional TLV of association setup:  Can be eliminated i HBI
   is a FP LFB attribute (as listed on pp.16.).

pp.24., Section 6.1.2: TLV can be avoided if result is encoded as part
   of PL-FLAGS.  A few bits should be sufficient, and the flags field
   has 32.

pp.25., Section 6.1.3, "Message transfer direction": "(or CE to CE)"
   Why?

pp.25., Figure 10: I don't think you mean that the reason field of the
   TLV is optional.  It is either that the TLV as such is optional or
   the Reason field has a special value encoding "Reason not
   specified". Which one is the case?

pp.30., Figures 13 and 14 (vs. Figure 11):  Implies that if N queries
   are requested, all responses must be sent in same response message.
   Is that the intent?  If yes, this is too rigid.  If not, then you
   will need a correlator in the PL header.  See comment #5 above.

pp.32., "Type (16 bits)": I expect this list to evolve.  If event
   subscription will be managed via attribute settings (which I think
   it should), then the Subscribe/Unsubscribe can be eliminated.
   "Del All" may be eventually implemented as special index pair used
   in a slice delete operation.  I assume this list will be refined as
   the model team finalizes "Config Data".

pp.32., Figure 16: "Event type" will not be needed if event
   subscription will be managed via LFB attributes.

pp.33.: What is the uniqueness scope of the "Instance ID"?

pp.35., Section 6.4.1: Can you give an example of a CE(-->FE) event
   (which is not a config message nor a session tear-down)?

pp.36., Figure 19, 'Type="FEEventNotification vs. CEEventNotification':
   Redundant.  Can be deduced from FE/CE addresses in PL header.

pp.38., Section 6.5: Packet Redirect Message:  I don't think this is
   a good name.  It does not cover the what may be the bulk of the
   CE-FE traffic: locally terminated traffic.

   Also: Please elaborate if this is the only way to send packets
   between the FE and CE or if other mechanisms are allowed.

pp.42., Figure 26: This message can be eliminated if modeled as FE LFB
   attribute.

Best regards,
--
Zsolt Haraszti                Phone:  +1 919-765-0027/2017
Modular Networks              Mobile:      +1 919-522-2337
                              Email:  [email-masked]

---

From: Robert Haas <[email-masked]>
Date: Thu, 5 Aug 2004 17:19:20 +0200

Zsolt,
Thanks for your detailed review and comments. As I am just catching up 
with email now, here are my first answers to two of your questions:

>   setting/querying the respective attribute(s) of the "FE LFB."
This is the method that I proposed for the first time in July 2003 at 
the Vienna IETF. Fully adopting it implies some more "purification", as 
you describe it. The current protocol draft has left this open for the 
WG to recommend whether it is the way to go. You obviously seem to like 
it ;-)

The protocol team also debated how much purification is tolerable. For 
instance, whether a PL Heartbeat message should be of its own type, or 
whether it should be treated just as any other LFB event message. We 
decided that a message type of its own was preferrable for Heartbeats, 
for simplicity and clarity reasons mostly, if I recall well.

>   If this is not clear, I am happy to describe this in more detail.
We discussed this briefly on the WG list on June 30, 2004. A group ID is 
used to multicast the PL message. The LFB ID would be left to a special 
reserved value. A 2-dimensional-array attribute in the FE Protocol 
Object would list:
- the groups to which an FE belongs to, and
- the corresponding LFBs in each of those groups.

I understand that this is different than the additional virtual LFB ID 
you propose. But given the huge amount of available group IDs, this can 
work too. Note that this means that a group ID conceptually groups LFBs 
together, across FEs.

Regards,

-- 
Robert Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-1-724-8698  fax +41-1-724-8578  http://www.zurich.ibm.com/~rha

---

From: Zsolt Haraszti <[email-masked]>
Date: Thu, 5 Aug 2004 12:17:59 -0400

> for simplicity and clarity reasons mostly, if I recall well.

I agree.  Heartbeats have to be simple, should not require
much processing.

> work too. Note that this means that a group ID conceptually groups LFBs

That would work too.  The solution I proposed divides the multicast
grouping into two layers: FEs can form multicast trees; LFBs inside
the FEs can form multicast trees.  The two layers appear independently
managed, but that is not entirely true.  They need to be managed in
sync, because there is no point to send the same LFB config request
to N number of FEs, unless the LFB ID in the message makes the same
sense to all those FEs.  So maybe the layering is artificial anyway.

Your solution handles the two layers as one.