Subject: [IPFIX] Fwd: Template Set padding issue

From: Brian Trammell <[email-masked]>
Date: Tue, 7 Nov 2006 15:06:42 -0800

All,

In ensuring last week that our own IPFIX implementation handles all  
the vagaries of set padding properly, I found what I think is an  
inconsistency in the protocol's handling of Template Set padding. The  
issue is best illustrated by the question, "what does a Collecting  
Process do with the following Template Set example?":

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 (Template Set)     | Set Length = 32               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 0xB701          | IE Count = 4                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| IE 1 = 9  s.ActiveHostCount | IE 1 Length = 4               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IE 1 PEN = 6871 (CERT Network Situational Awareness Group)    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| IE 2 = 44 s.IPv4Prefix      | IE 2 Length = 4               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| IE 3 = 9  s.IPv4PrefixLen.  | IE 3 Length = 1               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| IE 4 = 12 d.IPv4Address     | IE 4 Length = 4               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set padding to 8-byte boundary = 0                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

This is at first glance a valid template set, aligned to an 8-byte  
boundary as suggested in "Padding" in sec. 3.3.1 of proto-23. However:

1. The padding length MUST be shorter than any allowable Record in  
this Set. ("Padding", sec. 3.3.1 of proto-23); and

2. Template Withdrawal messages (sec 8 of proto-23) set the minimum  
allowable Template Record length to 4 bytes (a Template ID followed  
by an IE Count of 0).

So, if a Collecting Process were to follow the protocol (as I read it  
presently) and note that the remainder of the set after the first  
template was equal to the minimum Template Record length, it would  
interpret it as follows:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 (Template Set)     | Set Length = 32               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 0xB701          | IE Count = 4                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| IE 1 = 9  s.ActiveHostCount | IE 1 Length = 4               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IE 1 PEN = 6871 (CERT Network Situational Awareness Group)    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| IE 2 = 44 s.IPv4Prefix      | IE 2 Length = 4               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| IE 3 = 9  s.IPv4PrefixLen.  | IE 3 Length = 1               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| IE 4 = 12 d.IPv4Address     | IE 4 Length = 4               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 0x0000          | IE Count = 0                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

which would be an illegal Template Withdraw record (as such records  
can only appear in a message with other Template Withdraw records:  
"The Template Withdraw Message MUST NOT contain new Template or  
Options Template Records.", sec. 8 of proto-23) for an illegal  
Template ID (Set ID 0x0000 is reserved from V9, sec. 12 para 2 of  
proto-23; also RFC3954). So, any Template Set padded to an 8-byte  
boundary would appear to be an illegal Set.

One solution would be to explicitly point out that any Template  
Record which is a four-byte sequence of zeroes be interpreted as  
padding, as a special case. In other words, a Collecting Process  
would accept and ignore any Template Withdraw record for Template 0.

This solution would also imply that we would want to state that any  
Options Template Record which is four-byte sequence of zeroes (a  
Template Withdraw record as above) or a six-byte sequence of zeroes  
(which would be patently illegal as an Options Template Record) be  
interpreted as padding as well.

This does allow us to pad any type of set to 8-byte boundaries, but  
has the disadvantage of mandating two extra special-case codepaths at  
the CP side. Therefore, I'm a little conflicted about it as a  
proposal. Thoughts?

Regards,

Brian

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Tue, 7 Nov 2006 15:06:42 -0800

All,

Regards,

Brian

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 08 Nov 2006 08:42:24 +0000

Hi Brain

It seems to me that the either the withdrawal message is either
"allowable Record in this set" or it isn't.  So does the fact
that Template Withdrawal records can only appear in a set of
other Withdrawal messages mean that it isn't an "allowable
record"?

1. It is allowed:
  As you quote below "The padding length MUST be shorter than
  any allowable Record in this Set."  If this is an allowable
  record then you can only pad 3 or less bytes, therefore your
  below message is illegal as it wants to pad 4 bytes.

2. It is not allowed:
  If it's a message that is not "allowable" then the padding
  is valid and the record should be correctly decoded by the
  collector.

I think the spirit of the text is option 2.  The collector
should not be looking for withdrawal messages here.  I agree
that you've found a potentially sticky area, but I hope we
can clarify the case one way or the other.

Brian Trammell wrote:
> All,
> 
> In ensuring last week that our own IPFIX implementation handles all the 
> vagaries of set padding properly, I found what I think is an 
> 1. The padding length MUST be shorter than any allowable Record in this 
> Set. ("Padding", sec. 3.3.1 of proto-23); and
> 
> allowable Template Record length to 4 bytes (a Template ID followed by 
> an IE Count of 0).
> 
> which would be an illegal Template Withdraw record (as such records can 
> only appear in a message with other Template Withdraw records: "The 
> Template Withdraw Message MUST NOT contain new Template or Options 
> Template Records.", sec. 8 of proto-23) for an illegal Template ID (Set 
> ID 0x0000 is reserved from V9, sec. 12 para 2 of proto-23; also 
> RFC3954). So, any Template Set padded to an 8-byte boundary would appear 
> to be an illegal Set.
> 
> One solution would be to explicitly point out that any Template Record 
> which is a four-byte sequence of zeroes be interpreted as padding, as a 
> special case. In other words, a Collecting Process would accept and 
> ignore any Template Withdraw record for Template 0.
> 
> This does allow us to pad any type of set to 8-byte boundaries, but has 
> the disadvantage of mandating two extra special-case codepaths at the CP 
> side. Therefore, I'm a little conflicted about it as a proposal. Thoughts?
> 
> Regards,
> 
> Brian
> 
>

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 08 Nov 2006 08:42:24 +0000

Hi Brain

1. It is allowed:

2. It is not allowed:

---

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 8 Nov 2006 17:12:49 +0100

Dear Brian, Andrew,

If I am not mistaken here, you cannot mix templates with withdraw templates.

So (if this is the case) this is a non issue.

Regards,

Thomas

-- 
Thomas Dietz                       E-mail: [email-masked]
Network Laboratories               Phone:  +49 6221 4342-128
NEC Europe Ltd.                    Fax:    +49 6221 4342-155
Kurfuersten-Anlage 36
69115 Heidelberg, Germany          http://www.netlab.nec.de

> -----Original Message-----
> From: Andrew Johnson [mailto:[email-masked]] 
> Sent: Wednesday, November 08, 2006 12:42 AM
> To: Brian Trammell
> Cc: [email-masked]
> Subject: Re: [IPFIX] Fwd: Template Set padding issue
> 
> Hi Brain
>

---

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 8 Nov 2006 17:12:49 +0100

Dear Brian, Andrew,

Regards,

Thomas

---

From: Paul Aitken <[email-masked]>
Date: Wed, 08 Nov 2006 16:21:06 +0000

> If I am not mistaken here, you cannot mix templates with withdraw

Thomas is right: the fourth paragrah under Figure T in IPFIX-PROTO says:

    The Template Withdraw Message MUST NOT contain new Template or
    Options Template Records.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 08 Nov 2006 16:21:06 +0000

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Nov 2006 09:01:25 -0800

Indeed, the protocol does not allow Template Withdrawals other than  
in their own Set (and these in their own Message). So, the minimum  
allowable record length in a Set is then as follows:

Template Set: IF Template Withdrawal, then 4 (TID + 0x0000);
                 otherwise 8 (TID + IE Count + IE 0 + Len 0)

Options Template Set: IF Template Withdrawal, then 4 (TID + 0x0000);
                 otherwise 10 (TID + IE Count + Scope Count + IE 0 +  
Len 0)

So the question now becomes "how does the Collecting Process  
determine that a Template Set contains Template Withdrawals". Since  
the protocol doesn't define a Set ID for template withdrawals (the  
Set ID MUST be 2 to withdraw Templates or 3 to withdraw Options  
Templates), when reading the first {Options} Template Set in a  
Message it must be assumed to possibly contain a Template Withdrawal  
until the first Template Record is read.

This is actually not all that bad WRT requiring special case  
treatment at the CP (and is certainly much more acceptable than my  
earlier proposal). Padding will also work in this case (as padding is  
read at the _end_ of the Template Set, after the first record is  
read). However, as an implementor, I would still suggest that we note  
this is a bit tricky in the implementation guidelines...

Thanks!

- Brian

On Nov 8, 2006, at 12:42 AM, Andrew Johnson wrote:

>> In ensuring last week that our own IPFIX implementation handles  
>> all the vagaries of set padding properly, I found what I think is  
>> an inconsistency in the protocol's handling of Template Set  
>> padding. The issue is best illustrated by the question, "what does  
>> a Collecting Process do with the following Template Set example?":
>> 2. Template Withdrawal messages (sec 8 of proto-23) set the  
>> minimum allowable Template Record length to 4 bytes (a Template ID  
>> followed by an IE Count of 0).
>> So, if a Collecting Process were to follow the protocol (as I read  
>> it presently) and note that the remainder of the set after the  
>> first template was equal to the minimum Template Record length, it  
>> would interpret it as follows:
>> which would be an illegal Template Withdraw record (as such  
>> records can only appear in a message with other Template Withdraw  
>> records: "The Template Withdraw Message MUST NOT contain new  
>> Template or Options Template Records.", sec. 8 of proto-23) for an  
>> illegal Template ID (Set ID 0x0000 is reserved from V9, sec. 12  
>> para 2 of proto-23; also RFC3954). So, any Template Set padded to  
>> an 8-byte boundary would appear to be an illegal Set.
>> This solution would also imply that we would want to state that  
>> any Options Template Record which is four-byte sequence of zeroes  
>> (a Template Withdraw record as above) or a six-byte sequence of  
>> zeroes (which would be patently illegal as an Options Template  
>> Record) be interpreted as padding as well.
>> This does allow us to pad any type of set to 8-byte boundaries,  
>> but has the disadvantage of mandating two extra special-case  
>> codepaths at the CP side. Therefore, I'm a little conflicted about  
>> it as a proposal. Thoughts?
>> Regards,
>> Brian

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Nov 2006 09:01:25 -0800

Thanks!

- Brian

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 08 Nov 2006 18:16:21 +0000

Brian Trammell wrote:
> Indeed, the protocol does not allow Template Withdrawals other than in 
> their own Set (and these in their own Message). So, the minimum 
>                 otherwise 10 (TID + IE Count + Scope Count + IE 0 + Len 0)
> 
> So the question now becomes "how does the Collecting Process determine 
> that a Template Set contains Template Withdrawals". Since the protocol 
> doesn't define a Set ID for template withdrawals (the Set ID MUST be 2 
> to withdraw Templates or 3 to withdraw Options Templates), when reading 
> the first {Options} Template Set in a Message it must be assumed to 
> possibly contain a Template Withdrawal until the first Template Record 
> is read.

If the protocol says that the withdrawals must be in their own message,
them the decision can be made on the very first Set.  I would recommend
that on receiving a message, determining if it is a withdrawal message is
one of the first actions to be taken.

Can we have a message with both Template withdrawals and Option Template
withdrawals or do they have to be sent separately?

> This is actually not all that bad WRT requiring special case treatment 
> at the CP (and is certainly much more acceptable than my earlier 
> proposal). Padding will also work in this case (as padding is read at 
> the _end_ of the Template Set, after the first record is read). However, 
> as an implementor, I would still suggest that we note this is a bit 
> tricky in the implementation guidelines...

I agree.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 08 Nov 2006 18:16:21 +0000

I agree.

regards

Andrew

---

From: Paul Aitken <[email-masked]>
Date: Wed, 08 Nov 2006 18:38:22 +0000

Andrew,

I agree.

> withdrawals

I don't see why not.

> or do they have to be sent separately?

Is there any need for that?

Actually, it seems to me that "Template Withdrawal Message" is a 
misnomer: it's really a Template Withdrawal *Set*, and what's important 
is that you don't try to define new templates in the same set - else you 
might end up in the situation Brian described earlier.

I don't see a problem with sending both Template withdrawals and Option 
Template withdrawals as two sets in the same IPFIX message, nor with 
sending a mixture of templates and withdrawals in different sets in the 
same message.

I think it'd be a lot clearer if the "Template Withdrawal Message" was 
renamed to "Template Withdrawal Set" and this text:

was changed to:

    The Template Withdraw Set MUST NOT contain new Template or
    Options Template Records.

>> However, as an implementor, I would still suggest that we note this is 
>> a bit tricky in the implementation guidelines...
> 
> I agree.

Seconded.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 08 Nov 2006 18:38:22 +0000

Andrew,

I agree.

I don't see why not.

was changed to:

Seconded.

---

From: Benoit Claise <[email-masked]>
Date: Wed, 08 Nov 2006 21:22:34 -0800

Hi,
>    Options Template Records.
On the top of that, the Template Id must >255. So not zero.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Wed, 08 Nov 2006 21:22:34 -0800

Hi,

Regards, Benoit.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Nov 2006 21:42:18 -0800

On Nov 8, 2006, at 10:38 AM, Paul Aitken wrote:

> misnomer: it's really a Template Withdrawal *Set*, and what's  
> important is that you don't try to define new templates in the same  
> set - else you might end up in the situation Brian described earlier.
>
> I don't see a problem with sending both Template withdrawals and  
> Option Template withdrawals as two sets in the same IPFIX message,  
> nor with sending a mixture of templates and withdrawals in  
> different sets in the same message.
>
> I think it'd be a lot clearer if the "Template Withdrawal Message"  
> was renamed to "Template Withdrawal Set" and this text:
>

Absolutely. Benoit: are we at a state with the protocol now (sorry!)  
that we could make this change?

- Brian

---

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Nov 2006 21:42:18 -0800

- Brian