Subject: Re: Metadata model for the ForCES model

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 1 Feb 2004 10:14:30 -0500

Hi Zsolt,

Some comments:

+ I think with FE-FE metadata we are now treading into FE-FE
domain. I have no problem with this, at least the model related to the
metadata would be good to see in the draft. A lot of chip vendors (and
box vendors) already have protocols implementing interFE metadata
transfers as proprietary - they call them "stacking protocols"
(typically attached to a frame). While it may be too early to get
involved in such a beast, i think it should be put into consideration
(it is only mentioned in passing). I had a conversation on this in MPLS
with Alan Dekok and wouldnt mind coauthoring a draft with interested
people.

+ The first 3 sections are well written
Theres a claim that register based metadata is "more efficient". This
claim is not substantiated in any text. Also not sure why a scheme
on how metadata is _implemented_ needs to be mentioned in a model.
Metadata "moves" with the packet. Model shouldnt care how.

+ section 4 The Proposed Metadata Model

- I think a TLV with flags would be appropriately model metadata.
[tag = type; split the type bits into some 4 bits for flags]
I also think there is no reason to limit the length to 32 bits
hence the reason a TLV would be a good represantation.

+ section 5 Metadata Production and Consumption
"There may be multiple consumers for the same metadata"
is repeated twice.
- wasnt clear on the "conditional" aspect of metadata; could you provide
a real example?

+ section 7 Fixed Tag, Variable Tag, and Configurable Tag Metadata
Production

Isnt the tag selector itself metadata? I think starting this section
you may be treading into implementation specifics.
An LFB should be able to receive a number of metadata and ignore some of
them either based on LFB config or a metadata-selector passed with the
packet.

+section 9 Metadata Usage Categories

- "this value also is the value what the metadata  carries
between the LFBs"               ^^^^
----> should be "that"

- "Another way of setting  up  binding  relations  is  when  a
naturally occurring unique identifier of the consumer's object"
What is the event that would cause this? Is it a packet arrival or
a CE reading something off the FE?

- "is used as a reference (and as  a  value"
---> you need to close the brackets

- "definiton" typo

- "The value of the enumerated metadata may or may not  be  conveyed
via the ForCES protocol between the CE and FE."

Didnt grasp that.  How are LFBs on an FE supposed to cooperate on values
of metadata if they are not synced in this case?

cheers,
jamal

---

From: Zsolt Haraszti <[email-masked]>
Date: Tue, 3 Feb 2004 16:13:59 -0500

Jamal,

Thanks for your valuable comments.  See my replies embedded.

On Sun, 2004-02-01 at 10:14, Jamal Hadi Salim wrote:
> (it is only mentioned in passing).
Good comment.  The consideration I had was that we nail down an
LFB-to-LFB communication model which should be independent of
whether the communicating LFBs are on the same FE or on two or more
separate FEs.  This model does not specify how the communication
happens, it only says that if we want to define LFB classes and program
LFBs instance via the ForCES protocol, this is how we have to think
about passing per-packet state from one LFB to another.  So, so far
the model text leaves the actual implementation of LFB-to-LFB
communication completely open (well, the ForCES model does not ever
require that LFBs within an FE are  implemented as distinct processing
stages...).

Within an FE I expect that the vendors will always have infinite
freedom how the ForCES model is realized.

Between FEs's, however, the model does put some requirements on the
implementation.  At the minimum, it implies that a certain set of
information (i.e., the set of metadata that is present at the
exit point on FE1 and needed for further processing on FE2) must be
passed _in some way_ between the two FEs.  How it is passed is left
to the vendors at this point.  One can speculate that many vendors
would benefit from something less proprietary, so I expect that
ForCES will eventually start working on that area.  When the work
starts, it will have to take this metadata model into consideration.

> I had a conversation on this in MPLS
I'm sure it will be useful.  Let me know if I can help.

> Metadata "moves" with the packet. Model shouldnt care how.
OK, the text was written with the primary intent to drive the
metadata discussion to a conclusion (first within the model team,
then within the ForCES WG), and with the secondary intent to offer
text fragments to the new revision of the model draft.  But I expect
only a portion of this text will go into the revised draft.  Much of
the motivation/justification text, including the above, will probably
not make it.

> [tag = type; split the type bits into some 4 bits for flags]
Your thinking may be to protocol driven.  Again, this model--so far--
is mainly to describe a reference model w.r.t. how data passing should
be envisioned when defining LFBs, their attributes, and the CE-FE
protocol communication.  It is not about how to actually pass
metadata among LFBs.  For the above (reduced) scope, it has some
benefits to regard most metadata values as 32 bit integers.

BTW, what is the intended usage of the flags?

> hence the reason a TLV would be a good represantation.
See my comments above.

> - wasnt clear on the "conditional" aspect of metadata; could you provide
I do not have very strong examples to support this.  What I had
in mind was cases when the need for a metadata depends on the
value of another metadata.  For example, one metadata is a
RESULTKEY, and another is an ERRORCODE.  RESULTKEY is normally
used to encode the result of an operation (e.g., a search in
a table), and a special value (0) encodes the lack of result.
When RESULTKEY=0, then ERRORCODE tells more about the reason
of the miss.  When RESULTKEY!=0, then ERRORCODE is meaningless,
so it can be omitted.  I know that this can be done in other ways
(embedding the error codes into the RESULTKEY value space;
or making ERRORCODE mandatory, but ignore it in downstream
LFBs if RESULTKEY!=0, etc.)
Once we go through the exercise of creating a library of fully
defined LFBs and putting together useful configurations we may
conclude that there is no need for this feature.  Then we can pull
this out.  Until then, I left it in a space-holder.  But I can
be convinced...

> you may be treading into implementation specifics.
I am disconnected here.  Consider an Re-director LFB that can be
instantiated to split traffic into multiple streams based on some
input metadata.  For example, one instance splits based on a classifier
key (CLASSID), another splits based on COLOR, a third based on
protocolo type (PROTO).  This will be realized by an LFB class that
will have the following features (simplistic approach):
1) Variable number of outputs
2) An attribute that tells the LFB which metadata tag to look for
3) A table of entries, where each entry will contain a metadata
   value (as a condition), and a port reference (where to send
   the packet).
So the selector is an attribute of the LFB, read (or possibly written)
by the CE via the ForCES protocol.  The metadata values of the
table entries should be generic enough to accommodate (almost)
arbitrary metadata.  I see a benefit to make such a field in the
table entry a 32-bit integer field (which would serve 95% of
the expected metadata types based on our pre-study); that is what
motivates to have a generic data type for most metadata.

> them either based on LFB config or a metadata-selector passed with the
The LFB will use all metadata that is either specified
in the LFB specification (XML document) or referred by one of
the input metadata tag-selectors.  The latter is only applicable
to LFB classes that use variable/configurable tag selection. All
other metadata will be ignored by the LFB.

> a CE reading something off the FE?
Let me illustrate what I meant here with an example:
Lets say the consumer LFB has a table of things.  You may insert a
new entry in the table.  The operation returns with a table entry
index, which can be used in subsequent incremental table
management.  In the same time, the index can also be used as a
way to refer to the given entry from another LFB.  E.g, when an
upstream LFB wants to refer to one of the table entries, it can
send the index in a well-defined metadata.  That index is what I
consider "naturally occuring," meaning that it exists anyway
and it can also be used as a binding metadata.  So to answer the
original question: the event is a configuration event on the LFB
by the CE.

> of metadata if they are not synced in this case?
Consider a Rate Meter LFB that emits COLOR as metadata.  Consider
a downstream Rate Meter LFB that uses the COLOR metadata to do further
rate metering.  To neither of the LFB instances would we ever
communicate COLOR as part of the CE-FE communication.  It is all
built in in the specification of the LFB class and the specification
of the COLOR metadata.  So this is an example when the enumerated
metadata is not conveyed between the CE and FE.

Consider now the same Rate Meter LFB emitting the COLOR and a downstream
Redirector LFB set up to split the traffic into three paths based
on the COLOR.  When configuring the Redirector LFB, we would have to
tell the LFB to
1) Listen to COLOR metadata
2) Go to port 1, if COLOR=RED, port 2 if YELLOW, and port 3 if GREEN.
So in this case we would explicitly convey colors (COLOR values)
across the CE-FE interface.
> jamal
Thanks for taking your time for going through this.

/zsolt

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 3 Feb 2004 16:56:05 -0500

Interesting.  I had not read your meta-data description as addressing that.
I had assumed that the only reason for talking about FE-FE level meta-data
was to make clear it was out of scope.  Since packets leaving an FE are
presumed to be IP packets, I would expect normally want those to be just
plain vamilla IP packets.
Some implementations may use the available information to try to do
something extra.  That has been true of all routers.  But I would not
expect it to be within scope for ForCES.

Yours,
Joel M. Halpern

At 04:13 PM 2/3/2004 -0500, Zsolt Haraszti wrote:
>The consideration I had was that we nail down an

---

From: Steven Blake <[email-masked]>
Date: Tue, 3 Feb 2004 17:15:55 -0500

> expect it to be within scope for ForCES.

Ingress FE X receives packet, performs forwarding lookup, determines
packet should exit node on output interface Z on FE Y, via nexthop W.
Packet is forwarded to from FE X to Y.  How does FE Y determine Z & W?
o  Y performs duplicate forwarding lookup
o  Y uses metadata passed from FE X to convey Z & W.

Either case involves CE-FE configuration on Y.

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake               <[email-masked]>
Ericsson IP Infrastructure                +1 919-472-9913

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 3 Feb 2004 17:25:01 -0500

Given that the Fi reference point is explicitly outside of scope ("is not
currently defined by the ForCES architecture"), I had assumed that for now
the only choice was for FE Y to perform a duplicate lookup.
Or, to phrase it differently, I would say that under the current
environment FE X would determine that its next hop is FE Y, and FE Y would
determine that its next hop is W.  It is certainly true that this requires
CE configuration of FE X and FE Y.  But it does not require or expect any
metadata to be passed from FE X to FE Y.

In general, given that Fi is outside of scope, I think of all traffic to
another FE as if it were to another NE.

Yours,
Joel M. Halpern

---

From: [email-masked]
Date: Wed, 4 Feb 2004 10:13:57 +0900

> another FE as if it were to another NE.

Wouldn't that violate the rule that an NE, no matter how many parts it
is composed of, should appear to be a single entity.  If we assume that
FE-FE messaging is normal IP as if it were NE-NE then we may have
problems meeting that requirement.

I agree the current charter prohibits making FE-FE protocols a work
item, but I don't think that eliminates the need for implementations to
have a solution other then the passing of straight IP packets.
Including a metadata model that covers this seems like a good idea.
Personally I am glad someone is working on it.  I know I am more
concerned with the WG getting the model right then worrying about
specific protocol implementations.

a.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 3 Feb 2004 20:22:05 -0500

I do not believe that would violate single system appearence at all.
Appearing as a single system is a matter of management protocol
interactions and routing protocol advertisements.  It has nothing to do
with the packets flowing between FEs.

Yours,
Joel M. Halpern

---

From: [email-masked]
Date: Wed, 4 Feb 2004 11:04:17 +0900

Well, if IP is used between FEs, without any metadata in the exchange,
then wouldn't it involve  forwarding as it it were a hop?  And while it
is true that the receiving FE could treat the packet specially and not
modify the packet based on the extra hop, this would involve implicit
'metadata' that this wasn't being forwarded from outside the NE and not
as if between NEs.

Also, wouldn't there be a concern with the assignment of IP addresses
to each of the FEs.  Or would this be done based on some internal
private IP address space.

Maybe I don't really understand what you mean when you say an FE passes
a packet to another FE as if it were being passed to another NE.

a.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 3 Feb 2004 21:25:59 -0500

There probably is an issue of hop count.  Personally, I consider such
double decrementing a very good idea, as it avoids accidental internal loops.

Other than that, the IP addresses are just the interface IP addresses of
the NE.  If the links between the FEs are multi-access, they will need IP
addresses.  They will need them even if that network is otherwise
concealed, and even if the protocol were not IP.

One can craft all sorts of reasons why it might be nice to treat the Fi
links differently.  And later, we might choose to do so.  But the topology
and scope documents clearly state that we are not defining a protocol for
that.  And we can meet a reasonable definition of a "single system image"
without defining or using a special protocol on those links.
Given that the system works when we treat those links as carrying IP over
BACK (ie any technology used for the interconnect which supports IP
packets) it seems that we keep our work bounded and within defined scope if
for now we just use that.

Yours,
Joel M. Halpern

---

From: Zsolt Haraszti <[email-masked]>
Date: Tue, 3 Feb 2004 22:33:46 -0500

A couple of comments:

1. The fact is that most real distributed (multiblade) routers
    forward metadata from one blade (e.g., ingresss) to another
    (e.g., egress).  This simply allows optimal distribution of
    processing functions across the involved FEs.
    Not only that they forward metadata in addition to the IP packet
    being sent, but it may also be that the packet is not a
    complete/valid IP packet.  Since this is all within the node,
    it's all legitimate.

2. Just because it is not an IP packet or not only an IP packet,
    it does not mean that ForCES must include it in its scope.
    If it is not in the scope, so be it.  Then it will be left
    for the vendors, or to some other standards bodies to deal
    with it. (FYI, the NPF is addressing the inter-FE metadata
    encapsulation via a messaging LFB and messaging protocol).
    BTW, I think the ForCES work is still very useful, even if
    the inter-FE messaging is not in its scope.

3. Even if the inter-FE messaging could be put into the ForCES
    scope, we should still ignore it for the time being, and
    focus on the model and the CE-FE communication.

/zsolt

---

From: Alex Audu <[email-masked]>
Date: Wed, 4 Feb 2004 11:31:09 -0600

I am assuming this meta-data thing is all in context of having the CE
successfully
configure the FEs.  Unless this requires  FE-FE communication, I don't see any
reasons for extending this to include FE-FE.

Regards,
Alex.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 5 Feb 2004 11:46:27 -0500

> On Sun, 2004-02-01 at 10:14, Jamal Hadi Salim wrote:

[..]

> starts, it will have to take this metadata model into consideration.

Just to clarify what i was refering to is communication over something
like XAUI or CSIX inter-connected LFBs/FEs. It may get funky with path
MTUs with ethernet for example. I am also not about to change chip
vendors view of the world on already deployed products. However i see no
reason not to have the metadata carried along with the packet.
I think it will be a good idea you add text to at least emphasize things
in regards to LFBs across FEs. I agree on it being beyond ForCES as is
now.

> the motivation/justification text, including the above, will probably

When you make changes please post a diff -du of the two files. It is
easier to read. I think the idea of mentioning "registers" should be
left out of the text.

> benefits to regard most metadata values as 32 bit integers.

You have defined a Tag (that T) which takes a Value (that V).
I believe you could model better by not restricting the L to 4.

> BTW, what is the intended usage of the flags?

I was thinking you could carry the permissions for example such as
READ, WRITE etc; maybe they dont fit.

> LFBs if RESULTKEY!=0, etc.)

I see where you are going with it - it may be overkill. To me this
further strengthens the need to model based on TLVs. Clearly ERRORCODE.
and RESULTKEY are related; you are already refering to embedding them
together......

> this out.  Until then, I left it in a space-holder.  But I can

The conditionals are attributes of LFB topology; you may have taken
the approach that the topology definition needs metadata to route
the packet amongst LFBs (which is an implementation detail).
I think you should leave it in for now.

> by the CE via the ForCES protocol.

Making it an attribute is fine. I was more looking at the
view that the producer of the metadatum may also define the tag
selector and permissions as metadata which is consumed downstream.

> The metadata values of the

Ok, based on your statement above,  i see why i was having problems
with restricting it to 32-bit.
I would define metadata as "execution markers"; the only reason you need
metadata is so that you dont repeat work the packet has already been put
through. Using metadata for table lookups furher downstream is only a
subset of that domain.

To illustrate: (The linux kernel does this - apologies for going into imp.
its the best example i could think of that you could validate)
You could design it such that LFB execution completion could be deffered
when preempted by higher prio work. In this case you
may wanna later reenter the same LFB but skip some code. This is useful
in NPs as well. To achieve this you need to store in metadata not just
code pointers but data processed so far.

> original question: the event is a configuration event on the LFB

Reading all the above (and ignoring usage of metadata for a moment) it seems to me
you  are saying theres two types of metadata from a initial production pespective.
Theres  metadata that maybe we should call "static" that gets produced by the CE
as an LFB attribute. Theres also dynamic metadata that gets created or destroyed
by other events such as packet arrivals on the producer LFB or timers expiring
(eg a cache table entry creation/deletion).

> metadata is not conveyed between the CE and FE.

Right - but the attribute is still part of the LFB setup (via CE-FE protocol).
i.e from a LFB topology level theres still synchronization that is happening.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 5 Feb 2004 11:56:14 -0500

The problem (challenge actually) is this stuff exists already.
Looking the other way may not be the best option.
I agree with Zsolt's statements below.
If we want to say its outside the scope, then the text on metdata
that Zsolt posted is overkill already and may need a lot of chopping.

cheers,
jamal