Subject: Re: [sacm] IM Template Change Proposal

From: "Cokus, Michael S." <[email-masked]>
Date: Mon, 9 May 2016 18:41:10 +0000

> -----Original Message-----
> From: sacm [mailto:[email-masked]] On Behalf Of Jim Schaad
> Sent: Friday, April 08, 2016 10:11 AM
> To: [email-masked]
> Subject: [sacm] IM Template Change Proposal
>
> After the discussions in the last F2F as well as some side bar 
> meetings, I have some proposals of possible changes to the current 
> IPFIX template that we are using.

Thanks Jim.  I have some thoughts on this...

> 
> 1.  Delete the three container types in the current document.  
> (baseList, subTemplateList, subTemplateMultiList)

I agree and there seems to be a consensus forming that we need alternatives to the native IPFIX container types.  

> 
> 2.  Define two new container types
> 	* List - unordered list of items
> 	* OrderedList - ordered list of items

These make sense to me.  Using the IPFIX naming conventions, these would be called "list" and "orderedList".  Some initial  suggestions concerning words to describe them:

The list and orderedList types define named groups of IEs, and indicate whether to preserve the order of IEs in the list.
-The list type indicates that the IE order MAY be preserved.
-The orderedList type indicates that the IE order MUST be preserved.

So usage in an IE definition could look something like this:

personName: orderedList(lastName, firstName)

> 
> 3.  Define a new template field to hold the list definition for these 
> new types

Not sure I understand.  Is your suggestion to simply add a field to cover list/orderedList in the template specification (e.g. elementID, name, dataTYpe, etc.)?  I don't understand what the purpose of the new field would be.  Would it be a description of the list contents?

> 
> 4.  Define or modify the current list syntax to be, potentially, 
> somewhat richer.  I do not have a concrete proposal for this because I 
> am not sure what the set of properties that are needed here.  I 
> believe that this list includes one or more of the following:
> 
> *  The ability to have a set of properties

Not sure I understand.  What kind of properties are you talking about?  Or this
just a "bootstrap" for having the properties in the first place?

> *  The ability to define a property as optional, required, repeatable, 
> one of a list of options.

If I understand correctly, this pertains to IE cardinality.   I suggest the following:

Cardinality for IEs is indicated by:
"*" - zero or more occurrences
"+" - one to unbounded occurrences
"?" - zero or one occurrence
"(n,m)" - between n and m occurrences
The absence of cardinality indicator implies one occurrence.

So an example of usage in an IE definition could look something like this:

personName: orderedList(firstName, middleName?, lastName)

> *  There may be a requirement to be able to define sub-lists within the syntax.
> This is both a good thing to have and a potential problem.  If you 
> have something like  AddressSet ::= list( (addressType, ipv4Address | 
> ipv6Address |
> otherAddress) +) it is fine, but doing something like AddressSet ::= 
> (list (addressType, ipv4Address | ipv6Address), list(macAddress, 
> dnsAddress)) becomes a problem when trying to refer to the elements as 
> the lists are not themselves identified.  Sublists may not be worth 
> doing in order to have a name tag for each field.  It would be 
> possible however to basically create a syntax where the name is 
> supplied and the tree structure elements are implicitly created.  This makes descriptions slightly easier.
> 
> The easiest syntax to use is probably a standard RE type syntax.

I'm thinking it would best to disallow unnamed lists/sublists.  If not, it would be very complicated to identify/reference particular IE instances.  I know this is more related to the data models, but I think the naming needs to be in the IM to support it.

> Jim

Thanks,

--mike

Mike Cokus
Systems Engineer
The MITRE Corporation

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 9 May 2016 12:18:25 -0700

-----Original Message-----
From: Cokus, Michael S. [mailto:[email-masked]] 
Sent: Monday, May 09, 2016 11:41 AM
To: Jim Schaad <[email-masked]>; [email-masked]
Subject: RE: [sacm] IM Template Change Proposal

> (baseList, subTemplateList, subTemplateMultiList)

I agree and there seems to be a consensus forming that we need alternatives
to the native IPFIX container types.  

> 	* OrderedList - ordered list of items

These make sense to me.  Using the IPFIX naming conventions, these would be
called "list" and "orderedList".  Some initial  suggestions concerning words
to describe them:

The list and orderedList types define named groups of IEs, and indicate
whether to preserve the order of IEs in the list.
-The list type indicates that the IE order MAY be preserved.
-The orderedList type indicates that the IE order MUST be preserved.

> 3.  Define a new template field to hold the list definition for these 

Not sure I understand.  Is your suggestion to simply add a field to cover
list/orderedList in the template specification (e.g. elementID, name,
dataTYpe, etc.)?  I don't understand what the purpose of the new field would
be.  Would it be a description of the list contents?

[JLS] Looking at the current definition for softwareInstance (section 5.27
in version -04), the definition of the template for softwareInstance is
contained in the description field of the template.   I would propose that
this is moved from the description field to a new field. Thus you would get

elementId: TBD
name: softwareInstance
dataType: unorderedList
status: current
description: Information about an instance of software
                installed on an endpoint.

List:  unorderedList(allof,
                                    softwareIdentifier,
                                    title,
                                    creator,
                                    softwareVersion,
                                    lastUpdated
                                   )

> *  The ability to have a set of properties

Not sure I understand.  What kind of properties are you talking about?  Or
this just a "bootstrap" for having the properties in the first place?
[JLS] Trying to remember back, yes I believe that this is just a bootstrap
step.

> one of a list of options.

If I understand correctly, this pertains to IE cardinality.   I suggest the
following:

[JLS] I would make this an unordered list myself.  Given that the fields are
"tagged" there is no required order in the fields.  Ordered lists are setup
for things like indexing or associations between two different lists.  An
ordered list would be something like 

networkPath: orderedList(ipv4Address)

[JLS] This misses the one of a list.
softwareVersion = list( simpleVersion | rpmVersion | ciscoTrainVersion |
...)

[JLS] Also need to think about the question of extensibility.  Are lists
closed or open for additional elements.  I believe that the ... above is not
needed because we have an implicit statement that all lists can have
additional elements added.  The question would just be where they can be
added for ordered lists.

> The easiest syntax to use is probably a standard RE type syntax.

I'm thinking it would best to disallow unnamed lists/sublists.  If not, it
would be very complicated to identify/reference particular IE instances.  I
know this is more related to the data models, but I think the naming needs
to be in the IM to support it.

[JLS] Referencing is not a problem in the above.  It does require the
ability to index but you get something like
[{addressType:1, ipv4Address:1.2.3.4},{macAddress:XX.BB.CC,
dnsAddress:foo.bar.example.com},{addressType:2, ipv6Adress:0::0}]

Allowing for sublists is going to make the document shorter (potentially) as
new templates do not need to be defined for every structured element.
Common structured elements can be extracted as needed.

Jim

Thanks,

--mike

Mike Cokus
Systems Engineer
The MITRE Corporation

---

From: "Cokus, Michael S." <[email-masked]>
Date: Mon, 16 May 2016 18:48:01 +0000

Hi again.  More response/thoughts below:

> list/orderedList in the template specification (e.g. elementID, name, dataTYpe,
> etc.)?  I don't understand what the purpose of the new field would be.  Would it
> be a description of the list contents?
> 
> [JLS] Looking at the current definition for softwareInstance (section 5.27 in
> version -04), the definition of the template for softwareInstance is

[MSC] Ok, thanks.

> [JLS] Trying to remember back, yes I believe that this is just a bootstrap step.

[MSC] Ok, thanks.

> "tagged" there is no required order in the fields.  Ordered lists are setup for
> things like indexing or associations between two different lists.  An ordered list

[MSC] Agree.  In this case you wouldn't need to know the order to identify the
IEs used.

> would be something like

[MSC] I understand your point but wanted to make a quick side note on this.  Did you leave out the occurrence indicator on purpose?  If so then either + or * was implied, right?  Not sure we want this kind of shorthand.  But if we do, then we need to make sure it's clear what the implied meaning is.

> softwareVersion = list( simpleVersion | rpmVersion | ciscoTrainVersion |

[MSC] This brings up the question of which of the IPFX list semantics we want to keep.  It looks like oneOf is useful, but we could use regex "|" to represent this instead.  But borrowing  "|" from regex also
gives additional flexibility.  Are there other IPFIX list semantics we want to keep?  

> 
> [JLS] Also need to think about the question of extensibility.  Are lists closed or
> open for additional elements.  I believe that the ... above is not needed because
> we have an implicit statement that all lists can have additional elements added.
> The question would just be where they can be added for ordered lists.

[MSC] The open/closed list question hadn't occurred to me.  My implicit assumption was that the lists were closed.  Now that you bring it up, though, I agree that we need open lists.  

> 
> > *  There may be a requirement to be able to define sub-lists within
> > the
> syntax.
> I'm thinking it would best to disallow unnamed lists/sublists.  If not, it would be
> very complicated to identify/reference particular IE instances.  I know this is
> more related to the data models, but I think the naming needs to be in the IM to
> support it.
> 
> [JLS] Referencing is not a problem in the above.  It does require the ability to
> index but you get something like [{addressType:1,
> ipv4Address:1.2.3.4},{macAddress:XX.BB.CC,

[MSC] For the example above, I see your point.  But for more involved cases things become more complicated.  For instance, throw in optional elements/lists and unbounded lists into the mix.  With unnamed lists, I think we could need presence indicators.  In addition, I think having what are essentially multi-dimensional arrays requires additional information (i.e., row major versus column major) to insure interoperable data exchanges. 

> 
> Allowing for sublists is going to make the document shorter (potentially) as new
> templates do not need to be defined for every structured element.

[MSC] I wonder whether the benefit of sub-lists is worth the additional complexity.  Not saying that right now I know the full extent of the benefit/complexity trade off.  But I think it's worth discussing.

Thanks,

--mike

Mike Cokus
Systems Engineer
The MITRE Corporation

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 16 May 2016 12:33:08 -0700

-----Original Message-----
From: sacm [mailto:[email-masked]] On Behalf Of Cokus, Michael S.
Sent: Monday, May 16, 2016 11:48 AM
To: Jim Schaad <[email-masked]>; [email-masked]
Subject: Re: [sacm] IM Template Change Proposal

Hi again.  More response/thoughts below:

> > 3.  Define a new template field to hold the list definition for 
> > these new types
> 
> Not sure I understand.  Is your suggestion to simply add a field to 
> cover list/orderedList in the template specification (e.g. elementID, 
> name, dataTYpe, etc.)?  I don't understand what the purpose of the new 
> field would be.  Would it be a description of the list contents?
> 
> [JLS] Looking at the current definition for softwareInstance (section 
> 5.27 in version -04), the definition of the template for softwareInstance
is

[MSC] Ok, thanks.

> > somewhat richer.  I do not have a concrete proposal for this because 
> > I am not sure what the set of properties that are needed here.  I 
> Not sure I understand.  What kind of properties are you talking about?  
> Or this just a "bootstrap" for having the properties in the first place?
step.

[MSC] Ok, thanks.

> 
> 
> > *  The ability to define a property as optional, required, 
> > repeatable, one of a list of options.
> 
> [JLS] I would make this an unordered list myself.  Given that the 
> fields are "tagged" there is no required order in the fields.  Ordered 
> lists are setup for things like indexing or associations between two 
> different lists.  An ordered list

[MSC] Agree.  In this case you wouldn't need to know the order to identify
the IEs used.

> networkPath: orderedList(ipv4Address)

[MSC] I understand your point but wanted to make a quick side note on this.
Did you leave out the occurrence indicator on purpose?  If so then either +
or * was implied, right?  Not sure we want this kind of shorthand.  But if
we do, then we need to make sure it's clear what the implied meaning is.

[JLS] No I did not leave the indicator out on purposes, I just didn't think
about it. Part of this is the fact that I am thinking of two different
things when I say list.  There is the proposal that I am doing and the old
list that I said to kill.  In the old one then a single item is given and
the repeat indicator was implicit.  This is what I was thinking of when I
wrote this.  However, I believe that there should not be implied repetition
- so yes it is missing here.

> softwareVersion = list( simpleVersion | rpmVersion | ciscoTrainVersion 

[MSC] This brings up the question of which of the IPFX list semantics we
want to keep.  It looks like oneOf is useful, but we could use regex "|" to
represent this instead.  But borrowing  "|" from regex also gives additional
flexibility.  Are there other IPFIX list semantics we want to keep?  

[JLS] At this point I don't think so, but until we finish we probably don't
know for sure.

> 
> [JLS] Also need to think about the question of extensibility.  Are 
> lists closed or open for additional elements.  I believe that the ... 
> above is not needed because we have an implicit statement that all lists
can have additional elements added.
> The question would just be where they can be added for ordered lists.

[MSC] The open/closed list question hadn't occurred to me.  My implicit
assumption was that the lists were closed.  Now that you bring it up,
though, I agree that we need open lists.  

> > have something like  AddressSet ::= list( (addressType, ipv4Address 
> > | ipv6Address |
> > dnsAddress)) becomes a problem when trying to refer to the elements 
> > as the lists are not themselves identified.  Sublists may not be 
> > worth doing in order to have a name tag for each field.  It would be 
> > supplied and the tree structure elements are implicitly created.  
> > This
> makes descriptions slightly easier.
> >
> I'm thinking it would best to disallow unnamed lists/sublists.  If 
> not, it would be very complicated to identify/reference particular IE 
> instances.  I know this is more related to the data models, but I 
> think the naming needs to be in the IM to support it.
> 
> ability to index but you get something like [{addressType:1, 

[MSC] For the example above, I see your point.  But for more involved cases
things become more complicated.  For instance, throw in optional
elements/lists and unbounded lists into the mix.  With unnamed lists, I
think we could need presence indicators.  In addition, I think having what
are essentially multi-dimensional arrays requires additional information
(i.e., row major versus column major) to insure interoperable data
exchanges. 

> 
> Allowing for sublists is going to make the document shorter 
> (potentially) as new templates do not need to be defined for every
structured element.
> Common structured elements can be extracted as needed.

[MSC] I wonder whether the benefit of sub-lists is worth the additional
complexity.  Not saying that right now I know the full extent of the
benefit/complexity trade off.  But I think it's worth discussing.
[JLS] I think that I would start with allowing them and it would be "easy"
to remove them later if it looks to be too much of a mess.

Thanks,

--mike

Mike Cokus
Systems Engineer
The MITRE Corporation

_______________________________________________
sacm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/sacm

---

From: "Cokus, Michael S." <[email-masked]>
Date: Tue, 17 May 2016 19:55:37 +0000

Thanks for a good discussion Jim.  I encourage others in the group to weigh in.

> [MSC] This brings up the question of which of the IPFX list semantics we want to
> keep.  It looks like oneOf is useful, but we could use regex "|" to represent this
> instead.  But borrowing  "|" from regex also gives additional flexibility.  Are
> there other IPFIX list semantics we want to keep?
> 
> [JLS] At this point I don't think so, but until we finish we probably don't know for
> sure.

[MSC] As the group develops/proposes IE definitions, it should become more clear whether
we are missing something.  If anyone is/becomes aware of IPFIX list semantics [1] we
should think about keeping, please let the rest of the group know.

[1] https://tools.ietf.org/html/rfc6313#section-4.4

> assumption was that the lists were closed.  Now that you bring it up, though, I
> agree that we need open lists.

[MSC] It strikes that the rest of the group may not be clear about what 
open/extensible means in this context.  I won't speak for Jim, since we might
have actually have different ideas here.  To me, extensible means that you
can append IEs to a list which aren't specified in the list definition.  
Referring to Jim's example above, additional version IEs might 
be needed before they actually get added to the official definition of 
softwareVersion.  In this case, the flexibility to add/use it before going
through a complete cycle of configuration management would be handy.
We must be very careful about when/where this is allowed, how recipients
are supposed to interpret additional information which they may not
"understand" and what follow-up actions in the community are expected.  

> to remove them later if it looks to be too much of a mess.

[MSC] In the mean-time, I plan to play around with some examples to figure 
out what kinds of combinations may be problematic.
 
Thanks,
 
 --mike
 
Mike Cokus
Systems Engineer
The MITRE Corporation

---

From: "Jim Schaad" <[email-masked]>
Date: Tue, 17 May 2016 13:18:53 -0700

> Sent: Tuesday, May 17, 2016 12:56 PM
> > [MSC] This brings up the question of which of the IPFX list semantics
> > we want to keep.  It looks like oneOf is useful, but we could use
> > regex "|" to represent this instead.  But borrowing  "|" from regex
> > also gives additional flexibility.  Are there other IPFIX list semantics
we want to
> keep?
> >
> > [JLS] At this point I don't think so, but until we finish we probably
> > don't know for sure.
> 
> [MSC] As the group develops/proposes IE definitions, it should become more
> clear whether we are missing something.  If anyone is/becomes aware of
IPFIX
> list semantics [1] we should think about keeping, please let the rest of
the group
> know.
> > [MSC] The open/closed list question hadn't occurred to me.  My
> > implicit assumption was that the lists were closed.  Now that you
> > bring it up, though, I agree that we need open lists.
> 
can
> append IEs to a list which aren't specified in the list definition.
> Referring to Jim's example above, additional version IEs might be needed
before
> they actually get added to the official definition of softwareVersion.  In
this
> case, the flexibility to add/use it before going through a complete cycle
of
> configuration management would be handy.
are
> supposed to interpret additional information which they may not
"understand"
> and what follow-up actions in the community are expected.

[JLS]  I want to also point to the fact that this is going to need to be
supported for dealing with data models as well which extend the SACM IM in
some way.  That is they may add additional items to a list which need to  be
dealt with in some fashion.  Thus I think that the extensibility is forced
from a SACM requirement.

Jim

> > [MSC] I wonder whether the benefit of sub-lists is worth the
> > additional complexity.  Not saying that right now I know the full
> > extent of the benefit/complexity trade off.  But I think it's worth
discussing.
> [MSC] In the mean-time, I plan to play around with some examples to figure
out
> what kinds of combinations may be problematic.
> 
> Thanks,
> 
>  --mike
> 
> Mike Cokus
> Systems Engineer
> The MITRE Corporation

---

From: Adam Montville <[email-masked]>
Date: Thu, 19 May 2016 19:08:02 -0500

Do any others have opinions on this discussion?

> On May 17, 2016, at 3:18 PM, Jim Schaad <[email-masked]> wrote:
> 
> 
>