Subject: [ipfix] Message Source ID and template scoping

From: Brian Trammell <[email-masked]>
Date: Thu, 2 Mar 2006 00:12:30 +0100

Greetings,

In reviewing IPFIX-PROTO 19 during the interoperability event in  
Salzburg, I noticed what I believe to be an inconsistency in the  
definition of the Source ID message header field and the sourceId  
information element, and how these interact with template management.  
The relevant language:

(from 3.1, p. 13)
    Source ID
            A 32-bit identifier of the Observation Domain that is
            locally unique to the Exporting Process.  The Exporting
            Process uses the Source ID to uniquely identify to the
            Collecting Process the Observation Domain that metered the
            Flows.  Collecting Processes SHOULD use the combination the
            combination of the Exporter (exporterIPv4Address,
            exporterIPv6Address, or exportingProcessId) and the Source
            ID field to separate different export streams originating
            from the same Exporting Process.  The Source ID SHOULD be 0
            when no specific Source ID is relevant for the entire IPFIX
            Message.  For example, when exporting the Exporting Process
            Statistics, or in case of hierarchy of Collector when
            aggregated data records are exported.  The Source ID MUST be
            zero when the IPFIX Message contains data records with
            different Source ID values defined as scopes.

(from 3.4.1, p. 18)
     Template ID
          Each of the newly generated Template Records is given a unique
          Template ID.  This uniqueness is local to the Observation
          Domain that generated the Template ID.  Template IDs 0-255 are
          reserved for Template Sets, Options Template Sets, and other
          reserved Sets yet to be created.  Template IDs of Data Sets
          are numbered from 256 to 65535.  There are no constraints
          regarding the order of the Template ID allocation.

First, there's a typographical error (that may already be corrected  
in the working draft; I'm not sure) in the Source ID definition; on  
line 4: "the Collecting Process the Observation Domain" - should be  
"the Observation Domain".

These two paragraphs would seem to imply that the preferred way for  
Collecting Processes to manage templates is by (Source ID, Template  
ID) tuple (since Source ID is the observation domain identifier).  
However, this will lead to issues during aggregation, during which a  
single IPFIX Message may contain multiple Flow Records with different  
Source IDs (via Options Records or via the sourceId IE on each Flow  
Record). In this case, every _record_ is still scoped to a Source ID,  
but the templates no longer are. An exporter then has no way to  
ensure Template IDs associated with records from multiple upstream  
Observation Domains will not collide, and a collector accepting  
Source ID = 0 messages from multiple exporters may also run into  
Template ID collision problems. One way around this on the  
_collector_ side would be to manage templates by (Source ID, exporter 
(IPv(4,6)Address,ProcessId), Template ID) tuples, but this places  
further burdens on the collector to determine when two different  
Template IDs really are the same.

I would suggest instead that the Source ID of 0 be reserved* (using  
SHOULD NOT language); that _every_ IPFIX Message should have a valid  
(non-zero) Source ID; and that the source ID of a record is defined  
at the most specific of the enclosing message's Source ID, any Source  
ID attached to the record by an Options Record, then any Source ID  
attached directly to the record (via the sourceId IE).

The suggested new language is as follows:

    Source ID
            A 32-bit identifier of the Observation Domain that is
            locally unique to the Exporting Process.  The Exporting
            Process uses the Source ID to uniquely identify to the
            Collecting Process that metered the Flows.  Collecting  
Processes
            SHOULD use the combination the combination of the Exporter
            (exporterIPv4Address, exporterIPv6Address, or  
exportingProcessId)
            and the Source ID field to separate different export  
streams originating
            from the same Exporting Process.

	   The Source ID SHOULD NOT be 0. When no specific Source ID is  
relevant
	   for the entire IPFIX Message -- for example, when exporting the  
Exporting
	   Process Statistics, or in case of hierarchy of Collector when
            aggregated data records are exported -- the Source ID SHOULD
            identify the default Observation Domain for the Exporting  
Process,
	   or a virtual Observation Domain containing only the Exporting  
Process.
            The Source ID of each record MUST be taken to be the most  
specific
            Source ID applicable to the record.

* This is really something of an implementation convenience; it  
allows implementations which use IPFIX templates internally for  
record encoding and decoding to scope internal templates - those  
which will never leave the Process - by the (0, Template ID) source/ 
template tuple.

Thoughts?

- Brian

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 02 Mar 2006 10:19:37 +0100

Dear Brian, all,

I agree with your suggestion.

In my opinion, the Source ID should identify the Exporting Process and
every IPFIX messages sent by this Exporting Process has to contain its
Source ID. This is important for the collector in order to associate the
right Template(s). Thus, the collector can manage the Templates based on
the Source IDs only.

In case of a concentrator, the used Source ID would be the Source ID of
the Exporting Process within the concentrator. If you want to keep the
information about the original exporters, you can put this information
into the individual flow records or option records using the
exportingProcessId, exporterIPv6Address or exporterIPv4Address.

If a device runs more than one Exporting Processes, than it would use a
different Source ID for each. If we really need to export statistics
that are not specific to any Exporting Processes of a device, we could
do as Brian suggests: assign another Source ID to the device that is
exclusively used for that purpose.

Btw, I do not understand the following sentence in the definition of
Source ID:
"The Exporting Process uses the Source ID to uniquely identify to the
         Collecting Process that metered the Flows."
Does a Collecting Process meter flows?

Regards,
Gerhard

Brian Trammell wrote:
> First, there's a typographical error (that may already be corrected in
> the working draft; I'm not sure) in the Source ID definition; on line 4:
> "the Collecting Process the Observation Domain" - should be "the
> Observation Domain".
> 
> Collecting Processes to manage templates is by (Source ID, Template ID)
> tuple (since Source ID is the observation domain identifier). However,
> this will lead to issues during aggregation, during which a single IPFIX
> Message may contain multiple Flow Records with different Source IDs (via
> Options Records or via the sourceId IE on each Flow Record). In this
> case, every _record_ is still scoped to a Source ID, but the templates
> no longer are. An exporter then has no way to ensure Template IDs
> associated with records from multiple upstream Observation Domains will
> not collide, and a collector accepting Source ID = 0 messages from
> multiple exporters may also run into Template ID collision problems. One
> way around this on the _collector_ side would be to manage templates by
> (Source ID, exporter(IPv(4,6)Address,ProcessId), Template ID) tuples,
> but this places further burdens on the collector to determine when two
> different Template IDs really are the same.
> 
> (non-zero) Source ID; and that the source ID of a record is defined at
> the most specific of the enclosing message's Source ID, any Source ID
> attached to the record by an Options Record, then any Source ID attached
> directly to the record (via the sourceId IE).
> 
>            Collecting Process that metered the Flows.  Collecting Processes
>            and the Source ID field to separate different export streams
> originating
> * This is really something of an implementation convenience; it allows
> implementations which use IPFIX templates internally for record encoding
> and decoding to scope internal templates - those which will never leave
> the Process - by the (0, Template ID) source/template tuple.
> 
> Thoughts?
> 
> - Brian
> 

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Auf der Morgenstelle 10C 9P16, D-72076 Tuebingen, Germany
Phone: +49 7071 29-70534 / Fax: +49 7071 29-5220
EMail: [email-masked]
WWW:   http://net.informatik.uni-tuebingen.de/~muenz

---

From: Brian Trammell <[email-masked]>
Date: Fri, 3 Mar 2006 04:06:37 +0100

> Does a Collecting Process meter flows?

Nope - that was a typo. "the Collecting Process the Observation  
Domain that metered the flows" is in the original definition, and I  
deleted the wrong three words. The passage should read "The Exporting  
Process uses the Source ID to uniquely identify to the Observation  
Domain that metered the Flows."

Thanks,

Brian

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 04 Mar 2006 00:44:11 +0000

> right Template(s). Thus, the collector can manage the Templates based on

While that would be convenient, it would only work if all Exporting
Processes communicating with a single Collecting Process had different
Source IDs.  Since, in the majority of cases, different Exporting Process
will reside in different parts of the network and not communicate with
each other, this can never be guaranteed, which is why the use of the
source address is essential.

regards

Andrew

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Gerhard Muenz <[email-masked]>
Date: Sun, 05 Mar 2006 10:06:32 +0100

Dear Andrew,

Andrew Johnson wrote:
> source adress is essential.

Thanks for the clarification. I reread the current definition of
observation domains in IPFIX and it actually requires the (exporterIP,
sourceID)-tuple to identify an Exporting Process.

However, as you stated, the other option (unique sourceIDs for each
Exporting Process) would be convenient with respect to the
implementation of the collector. Moreover, it offers the advantage that
the same Exporting Process may use more than one interfaces (or IP
addresses) to export its monitoring data. According to the current
specification, no collector receiving data from two different
interface/IP addresses of the same Exporting Process would ever be able
to conclude that it receives duplicate records.

I assume that in practice, Exporting and Collection Processes would be
managed and configured by the same administrator, so it would not be a
problem to assign unique sourceIDs to the Exporting Processes.
But of course, there may have also been good resons to specify it as it
is today.

Regards,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Sun, 05 Mar 2006 10:20:38 +0000

Gerhard,

>sourceID)-tuple to identify an Exporting Process.
Collecting Processes SHOULD use the combination the 

>implementation of the collector.
>
But impossible to impliment, since all the exporters connected to the 
same collector would have to communicate with one another to agree 
unique IDs. Or they'd have to receive a unique ID from the collector 
before they can send any information.

However, if you think of "exporterIPv4Address : sourceID" as a 64bit 
value, you can consider it as a unique source ID for this exporter.

>Moreover, it offers the advantage that
>addresses) to export its monitoring data.
>
There's nothing to prevent that today.

>According to the current
Why not? Think of the exporting process as being at a level just above 
the interfaces, rather than being tied to one particular interface. The 
specification doesn't say to use the address of the interface, but to 
use the exporterIPv4Address / exporterIPv6Address, which is the 
IPv4/IPv6 address used by the Exporting Process. It's usually one of the 
interface addresses, but it could equally be another address that's 
unique to the box.

Cheers.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.