Subject: Re: [IPFIX] Per-SCTP stream draft: New Template ID SHOULD/MUST be used?

From: Brian Trammell <[email-masked]>
Date: Mon, 3 Mar 2008 14:32:57 -0500

Cool... I've seen the posting for the new revision, and will be  
reading it before I arrive in Philadelphia...

(I'm back from holidays, now...)

- Brian

On Feb 22, 2008, at 8:16 AM, Benoit Claise wrote:

> Thanks Brian,
>
> Let me quickly update the draft based on Gerhard's and your feedback.
>
> Regards, Benoit.
>> Benoit, Gerhard, and all,
>>
>> Replies inline; the most important one appears at the end, after  
>> Solution 1. Summary: we shouldn't change the protocol to redefine  
>> template scopes; it doesn't just break specific things but creates  
>> a new protocol; and, we should address dependent information  
>> sequencing at some point.
>>
>> On Feb 19, 2008, at 12:10 PM, Benoit Claise wrote:
>>
>>> Gerhard,
>>>
>>> You had a series of very good comments, which proves that we're  
>>> thinking along the same lines.
>>> For the everybody's sake, let me take a step back and explain the  
>>> problems we try to solve by sending template per-SCTP stream.
>>> This should answer some questions below, such as "What do you mean  
>>> by reducing the risk of losing Data Records?"
>>>
>>> Problem 1:
>>> - when a template ID is used by multiple streams, and a template  
>>> withdrawal message is sent on one of the those streams, then the  
>>> other streams must allow enough time for the template withdrawal  
>>> message to reach the collector before they can reuse the template  
>>> ID, which is something they cannot know.
>>> More importantly, the template withdrawal message cannot be sent  
>>> on the first stream until it knows that all the other streams have  
>>> transmitted their corresponding data records - which is also  
>>> impossible to know.
>>>
>>> It's true that RFC5101 specifies:
>>>   The Template Withdrawal Message MUST NOT be sent until sufficient
>>>   time has elapsed to allow the Collecting Process to receive and
>>>   process the last Data Record using this Template information.   
>>> This
>>>   time MUST be configurable.  A suitable default value is 5 seconds
>>>   after the last Data Record has been sent.
>>> But it's not always possible to guarantee that 5 seconds is  
>>> enough. So we might lose data records.
>>
>> Indeed. Template withdrawals (like UDP) are dangerous, and in an  
>> ideal situation should not be used at all. They won't work reliably  
>> in the general-case, flexible multistream environment. But  
>> sometimes you'll get yourself into a situation where you need to  
>> withdraw templates, for whatever reason. So we give guidelines, as  
>> above, to minimize the damage should an exporter choose to use this  
>> feature.
>>
>> (Aside: if data record loss avoidance is a prime requirement, use  
>> TCP or single-stream reliable SCTP.)
>>
>> (Aside, aside: Can someone provide 1. the count of templates you're  
>> expecting to use within a single transport session and 2. example  
>> expected use cases for template withdrawals? In theory all this is  
>> broken, I agree, but I still can't really see situations where all  
>> these various template withdrawal problems will arise in the Real  
>> World (tm).)
>>
>>> More generic problem:
>>> Problems 1 and 2 are different problems, but with the same cause:  
>>> the problems come from data records from the same template ID sent  
>>> across different streams. Typically, the PSAMP option template  
>>> record for sampling.
>>
>> More generically still, the problems come from dependencies between  
>> IPFIX messages sent without sequencing guarantees. See comment on  
>> Solution 3 below.
>>
>>> This is a current limitation/problem with IPFIX
>>> We evaluated whether we can propose some improvements part of this  
>>> per-SCTP draft, even if the per-SCTP draft goal was to export one  
>>> template per stream to be able to deduce the data loss for each  
>>> template record.
>>>
>>> There are a couple of solutions, with the solution 1 being the  
>>> preferred one. Hence the reverse order.
>>>
>>> Solution 3:
>>> As explained by Gehrard, look at the export time in the template  
>>> records, data records, and template withdrawal message.
>>> - This implies a more complex collecting process implementation as  
>>> the old and new template records must kept to compare the time
>>>  Note: potentially more than two template records must be kept if  
>>> the definition of the template ID changes frequently
>>> - This implies that we loose the real-time decode, as the  
>>> collecting process must wait for a certain time, just in case a  
>>> delayed template withdrawal message and a new template record is  
>>> received. For this reason only, I think this solution is poor
>>> - What if data record and template withdraw message were in the  
>>> same IPFIX message? Probably it'd depend on the order within the  
>>> message.
>>
>> The CP should be looking at export time when evaluating information  
>> dependencies (scoped options, templates), anyway. This is  
>> completely ignored by the protocol (though in retrospect perhaps it  
>> should have been), but there are really only two temporal scoping  
>> policies that make sense:
>>
>> 1. If dependencies cannot be withdrawn, then it can be held to be  
>> valid for the entirety of its enclosing outer scope (the transport  
>> session). In this case, ordering within the session is completely  
>> unimportant. This is pretty much, by default, how 5101 treats  
>> things, and I think how we think about these things on the list.
>>
>> 2. If dependencies may we withdrawn _AND REDEFINED_ (which is the  
>> point of template withdrawals, after all), then the ordering of  
>> these operations is critically important. Since ordering is  
>> important _across_ streams, we can't use the sequence number. The  
>> only sync signal we have that works across all streams is the  
>> export time.
>>
>> Actually building out a consistent set of rules for handling  
>> dependent information could, I think, be done without changing the  
>> wire protocol or the EP at all; it could basically be a set of CP  
>> guidelines. Though I haven't thought all the way through the  
>> problem to be able to say that with much confidence, yet. IIRC  
>> Andrew and I discussed something along these lines in the form of a  
>> "scope user's guide" way back in San Diego, but haven't had a  
>> chance to do anything with it since then.
>>
>>> Solution 2
>>> We could propose a series of specifications that would reduce the  
>>> likelihood that problem 1 or 2 would happen
>>>    - transmission order is maintained within a stream
>>>    - redefinition of the Template ID is not encouraged
>>>    - template withdrawal message SHOULD be sent in the last used  
>>> stream that carried the template ID to be removed
>>>    - etc..
>>> The drawback of this solution is that, even if the likelihood is  
>>> reduced, there is no guarantee that the problem 1 and 2 will not  
>>> happen.
>>> When I posted the initial email from this thread, I was still  
>>> thinking along this solution 2
>>
>> Something along the lines of this solution is, I think, as good as  
>> we can do, in view of the issues with solution 1; see below.
>>
>> An EP using the per-stream technique will not lose data records due  
>> to template withdrawals or template synchronization issues. This is  
>> an advantage of using per-stream. But per-stream is an option; a  
>> technique applicable to certain use cases that has specific  
>> advantages in those cases. It uses each template only on a single  
>> stream, and each stream has only a single template, but that is  
>> simply the EP's prerogative, not a property of the protocol.
>>
>> An EP not using template withdrawals at all will not lose data  
>> records due to template withdrawals, trivially.
>>
>> An EP choosing to use template withdrawals and cross-stream  
>> template definitions may generate a message stream it is difficult  
>> but not impossible for the CP to process without record loss. 5101  
>> provides guidelines and warnings to this effect already.
>>
>> (Again, data loss avoidance is _one_ requirement -- one policy --  
>> that the mechanisms of the IPFIX protocol can be used to implement.  
>> It is not always of the utmost importance; otherwise we wouldn't  
>> support UDP and PR-SCTP.)
>>
>>> Solution 1
>>> The template IDs are scoped per stream, next to the the Transport  
>>> Session and Observation Domain that generated the Template ID.
>>> Next to that, the transmission order per stream is maintained
>>> In this case, problem 1 and problem 2 will never happen
>>> The drawbacks are that
>>>    - more template ID and records would be usedFrom [email-masked]  Mon Mar  3 11:35:52 2008
From: Brian Trammell <[email-masked]>
To: Benoit Claise <[email-masked]>
Date: Mon, 3 Mar 2008 14:32:57 -0500
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [IPFIX] Per-SCTP stream draft: New Template ID SHOULD/MUST be
	used?
Sender: [email-masked]

Cool... I've seen the posting for the new revision, and will be  
reading it before I arrive in Philadelphia...

(I'm back from holidays, now...)

- Brian

On Feb 22, 2008, at 8:16 AM, Benoit Claise wrote:

> Thanks Brian,
>
> Let me quickly update the draft based on Gerhard's and your feedback.
>
> Regards, Benoit.
>> Benoit, Gerhard, and all,
>>
>> Replies inline; the most important one appears at the end, after  
>> Solution 1. Summar. For example, the  
>>> sampling option template record would require
>>>      one template ID per stream and the export the "same" data  
>>> record values per stream
>>>    - "reducing redundancy" draft would not work across stream. Is  
>>> this a big issue, I don't know!
>>
>> Indeed, this would cause options template records used for other  
>> than MP/EP metadata export (as in RFC5103, ipfix-exporting-type,  
>> ipfix-reducing-redundancy) not to work across streams.
>>
>> I agree with Gerhard (in his reply to this message): we shouldn't  
>> change the protocol. And this proposal does define a new protocol.  
>> RFC 5103 EPs would be incapable of interoperating with CPs that  
>> failed to interpret cross-stream templates. 5103 CPs would be  
>> incapable of interoperating with EPs that caused template collision.
>>
>> I would also repeat strenuously my warnings from the last time we  
>> visited this proposal (around the Chicago meeting last year) not to  
>> introduce restrictions or changes into the protocol that apply only  
>> to certain use cases and policies implementable with the greater  
>> protocol.
>>
>>> Note that there is a simple solution to address those two  
>>> drawbacks: never send a template withdrawal message, and make  
>>> those template IDs permanent as long as the SCTP association is  
>>> up. Nothing is changed on the collector side specifications, as it  
>>> doesn't have to know that the template IDs are permanent
>>>
>>> Now, the real question in this solution 1 is: what to do if we  
>>> can't increase anymore the number of streams in the SCTP  
>>> association?
>>> We export all the newly created template IDs and the associated  
>>> data records on the stream 0 (as currently specified in the per- 
>>> SCTP stream)
>>> Overloading the stream 0 is not an issue as the SCTP design is  
>>> such that there is one pool of buffer and all streams will share  
>>> it, i.e. "if you send  100k in one stream.. or 100k spread over  
>>> all streams.. it makes no difference" dixit Randy Stewart.
>>> Note: stream 0 is mentioned for historical but could very well be  
>>> another one. I'm not even sure we even need to specify a certain  
>>> stream
>>> Note that on that stream 0, we can't deduce the loss per template.
>>
>> Hmm. per-stream should also provide guidelines that it works best  
>> in cases where you have enough streams to handle all your templates.
>>
>> Cheers,
>>
>> Brian
>

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

y: we shouldn't change the protocol to redefine  
>> template scopes; it doesn't just break specific things but creates  
>> a new protocol; and, we should address dependent information  
>> sequencing at some point.
>>
>> On Feb 19, 2008, at 12:10 PM, Benoit Claise wrote:
>>
>>> Gerhard,
>>>
>>> You had a series of very good comments, which proves that we're  
>>> thinking along the same lines.
>>> For the everybody's sake, let me take a step back and explain the  
>>> problems we try to solve by sending template per-SCTP stream.
>>> This should answer some questions below, such as "What do you mean  
>>> by reducing the risk of losing Data Records?"
>>>
>>> Problem 1:
>>> - when a template ID is used by multiple streams, and a template  
>>> withdrawal message is sent on one of the those streams, then the  
>>> other streams must allow enough time for the template withdrawal  
>>> message to reach the collector before they can reuse the template  
>>> ID, which is something they cannot know.
>>> More importantly, the template withdrawal message cannot be sent  
>>> on the first stream until it knows that all the other streams have  
>>> transmitted their corresponding data records - which is also  
>>> impossible to know.
>>>
>>> It's true that RFC5101 specifies:
>>>   The Template Withdrawal Message MUST NOT be sent until sufficient
>>>   time has elapsed to allow the Collecting Process to receive and
>>>   process the last Data Record using this Template information.   
>>> This
>>>   time MUST be configurable.  A suitable default value is 5 seconds
>>>   after the last Data Record has been sent.
>>> But it's not always possible to guarantee that 5 seconds is  
>>> enough. So we might lose data records.
>>
>> Indeed. Template withdrawals (like UDP) are dangerous, and in an  
>> ideal situation should not be used at all. They won't work reliably  
>> in the general-case, flexible multistream environment. But  
>> sometimes you'll get yourself into a situation where you need to  
>> withdraw templates, for whatever reason. So we give guidelines, as  
>> above, to minimize the damage should an exporter choose to use this  
>> feature.
>>
>> (Aside: if data record loss avoidance is a prime requirement, use  
>> TCP or single-stream reliable SCTP.)
>>
>> (Aside, aside: Can someone provide 1. the count of templates you're  
>> expecting to use within a single transport session and 2. example  
>> expected use cases for template withdrawals? In theory all this is  
>> broken, I agree, but I still can't really see situations where all  
>> these various template withdrawal problems will arise in the Real  
>> World (tm).)
>>
>>> More generic problem:
>>> Problems 1 and 2 are different problems, but with the same cause:  
>>> the problems come from data records from the same template ID sent  
>>> across different streams. Typically, the PSAMP option template  
>>> record for sampling.
>>
>> More generically still, the problems come from dependencies between  
>> IPFIX messages sent without sequencing guarantees. See comment on  
>> Solution 3 below.
>>
>>> This is a current limitation/problem with IPFIX
>>> We evaluated whether we can propose some improvements part of this  
>>> per-SCTP draft, even if the per-SCTP draft goal was to export one  
>>> template per stream to be able to deduce the data loss for each  
>>> template record.
>>>
>>> There are a couple of solutions, with the solution 1 being the  
>>> preferred one. Hence the reverse order.
>>>
>>> Solution 3:
>>> As explained by Gehrard, look at the export time in the template  
>>> records, data records, and template withdrawal message.
>>> - This implies a more complex collecting process implementation as  
>>> the old and new template records must kept to compare the time
>>>  Note: potentially more than two template records must be kept if  
>>> the definition of the template ID changes frequently
>>> - This implies that we loose the real-time decode, as the  
>>> collecting process must wait for a certain time, just in case a  
>>> delayed template withdrawal message and a new template record is  
>>> received. For this reason only, I think this solution is poor
>>> - What if data record and template withdraw message were in the  
>>> same IPFIX message? Probably it'd depend on the order within the  
>>> message.
>>
>> The CP should be looking at export time when evaluating information  
>> dependencies (scoped options, templates), anyway. This is  
>> completely ignored by the protocol (though in retrospect perhaps it  
>> should have been), but there are really only two temporal scoping  
>> policies that make sense:
>>
>> 1. If dependencies cannot be withdrawn, then it can be held to be  
>> valid for the entirety of its enclosing outer scope (the transport  
>> session). In this case, ordering within the session is completely  
>> unimportant. This is pretty much, by default, how 5101 treats  
>> things, and I think how we think about these things on the list.
>>
>> 2. If dependencies may we withdrawn _AND REDEFINED_ (which is the  
>> point of template withdrawals, after all), then the ordering of  
>> these operations is critically important. Since ordering is  
>> important _across_ streams, we can't use the sequence number. The  
>> only sync signal we have that works across all streams is the  
>> export time.
>>
>> Actually building out a consistent set of rules for handling  
>> dependent information could, I think, be done without changing the  
>> wire protocol or the EP at all; it could basically be a set of CP  
>> guidelines. Though I haven't thought all the way through the  
>> problem to be able to say that with much confidence, yet. IIRC  
>> Andrew and I discussed something along these lines in the form of a  
>> "scope user's guide" way back in San Diego, but haven't had a  
>> chance to do anything with it since then.
>>
>>> Solution 2
>>> We could propose a series of specifications that would reduce the  
>>> likelihood that problem 1 or 2 would happen
>>>    - transmission order is maintained within a stream
>>>    - redefinition of the Template ID is not encouraged
>>>    - template withdrawal message SHOULD be sent in the last used  
>>> stream that carried the template ID to be removed
>>>    - etc..
>>> The drawback of this solution is that, even if the likelihood is  
>>> reduced, there is no guarantee that the problem 1 and 2 will not  
>>> happen.
>>> When I posted the initial email from this thread, I was still  
>>> thinking along this solution 2
>>
>> Something along the lines of this solution is, I think, as good as  
>> we can do, in view of the issues with solution 1; see below.
>>
>> An EP using the per-stream technique will not lose data records due  
>> to template withdrawals or template synchronization issues. This is  
>> an advantage of using per-stream. But per-stream is an option; a  
>> technique applicable to certain use cases that has specific  
>> advantages in those cases. It uses each template only on a single  
>> stream, and each stream has only a single template, but that is  
>> simply the EP's prerogative, not a property of the protocol.
>>
>> An EP not using template withdrawals at all will not lose data  
>> records due to template withdrawals, trivially.
>>
>> An EP choosing to use template withdrawals and cross-stream  
>> template definitions may generate a message stream it is difficult  
>> but not impossible for the CP to process without record loss. 5101  
>> provides guidelines and warnings to this effect already.
>>
>> (Again, data loss avoidance is _one_ requirement -- one policy --  
>> that the mechanisms of the IPFIX protocol can be used to implement.  
>> It is not always of the utmost importance; otherwise we wouldn't  
>> support UDP and PR-SCTP.)
>>
>>> Solution 1
>>> The template IDs are scoped per stream, next to the the Transport  
>>> Session and Observation Domain that generated the Template ID.
>>> Next to that, the transmission order per stream is maintained
>>> In this case, problem 1 and problem 2 will never happen
>>> The drawbacks are that
>>>    - more template ID and records would be used. For example, the  
>>> sampling option template record would require
>>>      one template ID per stream and the export the "same" data  
>>> record values per stream
>>>    - "reducing redundancy" draft would not work across stream. Is  
>>> this a big issue, I don't know!
>>
>> Indeed, this would cause options template records used for other  
>> than MP/EP metadata export (as in RFC5103, ipfix-exporting-type,  
>> ipfix-reducing-redundancy) not to work across streams.
>>
>> I agree with Gerhard (in his reply to this message): we shouldn't  
>> change the protocol. And this proposal does define a new protocol.  
>> RFC 5103 EPs would be incapable of interoperating with CPs that  
>> failed to interpret cross-stream templates. 5103 CPs would be  
>> incapable of interoperating with EPs that caused template collision.
>>
>> I would also repeat strenuously my warnings from the last time we  
>> visited this proposal (around the Chicago meeting last year) not to  
>> introduce restrictions or changes into the protocol that apply only  
>> to certain use cases and policies implementable with the greater  
>> protocol.
>>
>>> Note that there is a simple solution to address those two  
>>> drawbacks: never send a template withdrawal message, and make  
>>> those template IDs permanent as long as the SCTP association is  
>>> up. Nothing is changed on the collector side specifications, as it  
>>> doesn't have to know that the template IDs are permanent
>>>
>>> Now, the real question in this solution 1 is: what to do if we  
>>> can't increase anymore the number of streams in the SCTP  
>>> association?
>>> We export all the newly created template IDs and the associated  
>>> data records on the stream 0 (as currently specified in the per- 
>>> SCTP stream)
>>> Overloading the stream 0 is not an issue as the SCTP design is  
>>> such that there is one pool of buffer and all streams will share  
>>> it, i.e. "if you send  100k in one stream.. or 100k spread over  
>>> all streams.. it makes no difference" dixit Randy Stewart.
>>> Note: stream 0 is mentioned for historical but could very well be  
>>> another one. I'm not even sure we even need to specify a certain  
>>> stream
>>> Note that on that stream 0, we can't deduce the loss per template.
>>
>> Hmm. per-stream should also provide guidelines that it works best  
>> in cases where you have enough streams to handle all your templates.
>>
>> Cheers,
>>
>> Brian
>

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Mon, 3 Mar 2008 14:56:01 -0500

On Feb 21, 2008, at 7:31 PM, Paul Aitken wrote:

> Brian,
>> completely ignored by the protocol
>
> If you do that, then you'd have to store all the received data records
> undecoded for a certain amount of time - even if you have a valid
> template to decode them - against the slight possibility of a TWM and
> new template definition with an earlier timestamp being received from
> another stream.
>
> Which means you lose the ability to decode data in real time.
>

Yes, but note that you lose the ability to decode data in real time if  
and only if zero information loss in an environment in which  
withdrawals and dynamic template allocation are in use with multiple  
arbitrary streams is an inviolable requirement.

>> IIRC Andrew and I discussed something along these lines in the form  
>> of a "scope user's guide" way back in San Diego
>
> If templates contained a valid-from and a valid-to timestamp, with a
> guarantee they'd not be withdrawn or redefined within that window,  
> then
> you could decode in real time once again.

True indeed. Of course, right now the only way to add such information  
to the templates is via options, so there's still a bit of a  
bootstrapping problem here...

Cheers,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Mon, 3 Mar 2008 14:56:01 -0500

Cheers,

Brian