Subject: Re: [sacm] OVAL/SACM IM mapping document posted

From: "Hansbury, Matt" <[email-masked]>
Date: Wed, 3 Jun 2015 14:39:20 +0000

Hi David,

Thanks for reviewing the paper and the feedback.  Sorry for the delay in my response.   Comments inline.

Matt

From: David Ries [mailto:[email-masked]]
Sent: Thursday, May 21, 2015 4:03 PM
To: Hansbury, Matt
Cc: [email-masked]
Subject: Re: [sacm] OVAL/SACM IM mapping document posted

Hi Matt,

I second Gunnar’s praise of your work here. It does generally seem spot-on to me as well.

I do have questions about two small points raised in the document that I’d love some feedback on.

1) Platform Schema Extension Challenges
4.5.  Collection Guidance: “Important Note: One of the key drawbacks to OVAL is that ... is not easy or scalable to create or update extensions for rapidly changing platforms and products in a timely manner.”

Doesn’t the NETCONF schema demonstrate that it is not always necessary to create a platform extension to support a new or modified platform? Is there any language-based obstacle to adding more protocol-oriented schemas if that is desirable?

When it is desirable to create or modify a platform-specific schema, I’ve found that the primary obstacle is speed of the review and release process. And, the community seems to be taking steps to change that. For example, the recently revised versioning policy now allows platforms to be rev’d independently of the core so they can be released more quickly.

MJH:  Let me begin with I’m no NETCONF expert.   That said, if I’m following you, you’re suggesting by leveraging existing protocols like NETCONF (I would think WMI and PowerShell would be other examples here), we would be able to avoid the need for adding new tests to the language to support new assessment needs.  I don’t disagree, though I think there are a few points to consider along with this.  For one, this only works for the cases where we want to add functionality that is supported by the underlying protocol.  That doesn’t it mean it can’t be a useful tool here, but we probably do need to be prepared to address new concepts on some regular basis anyway, won’t we?

Second, recall that the addition of things like NETCONF and PowerShell tests to OVAL were not without some controversy.  Some were concerned that by providing access to these types of protocols, it made it harder to ensure and/or trust that OVAL-supported tools remain readonly.  To answer your specific question, there is no language-based obstacle to adding more protocol-based tests, but there may be non-technical reasons that we may need to consider.

Finally, I completely agree that the new policy of versioning the language extensions separately helps aid in limiting obstacles with respect to the release cycle.  I think that is consistent with our message to empower those that best know the products and platforms.  Ideally, the primary source vendor (or other similarly knowledgeable SME) is able to effectively articulate how to assess their products and is able to do so in a timely manner.

2) Collection & Evaluation Must Be De-Coupled
6.2.1.  Lesson: “the close coupling found in OVAL between the OVAL Object and OVAL State ... does not easily allow for dynamic querying of previously collected data, nor does it easily allow for efficiencies in data collection.”

I’ve heard this a few times in discussions of OVAL, but I’ve never really understood the criticism. It seems like this might be a misconception based on specific implementations of OVAL-based solutions, but not actually a limitation of OVAL itself.

I see an OVAL document as a policy statement of sorts and not a high-level programming language. The document specifies data that needs to be collected and (optionally) makes assertions about that data. But, is there a requirement to “run the whole document” in one go when a scan is performed?

To be more specific, I don’t see any obstacle to an enterprise solution storing the collected data and re-querying it later. Can’t you create definitions designed to collect posture attributes that include objects without states and store the collected data? Inventory definitions are another example of using OVAL to collect data that (presumably) is stored and queried later.

Additionally, I don’t see inherent obstacles to optimizing data collection for multiple objects. Is there a requirement that each object be collected individually? Can’t the collector analyze the data required by each of the objects contained in the superset of definitions applicable to a managed device and identify strategies for optimizing collection? For example, if there’s an object that specifies files in directory X and another that specifies files in directories X and Y, I’d expect the collector to only scan directory X once.

To give a slightly more complete example, an OVAL-based enterprise solution could do something like this:

  *   Retrieve large numbers of OVAL definitions (i.e. ~25k vulnerability and inventory definitions from Mitre, the Cisco IOS advisory definitions from Cisco, etc.)
  *   Analyze the definitions applicable to each platform in order to identify all of the data that must be collected from a managed device in order to resolve all the variables, generate items, etc. and evaluate the applicable definitons
  *   Collect and store that data
  *   Refresh that data when required by the organizations requirements for data freshness
  *   Evaluate any or all of the definitions on a schedule completely disconnected from the collection schedule

In such a solution, evaluating new or modified definitions may be possible based entirely on previously collected data.

MJH:  You are correct that there is nothing about OVAL that stops implementers from doing smart things during data collection, David.  We would certainly agree that OVAL, as it currently is, could be used to support such a scenario as you laid out.  However, because OVAL Objects (the ‘what to collect’ part) and OVAL States (the ‘what the data should look like’ part) are tightly coupled within the OVAL Definitions model, this scenario is not as easily supported as it could be.  Any data collected must be part of an OVAL Definition, along with a corresponding OVAL State, which means that you can’t proactively collect all security relevant configuration information, without the corresponding what it should look like part.  In SACM terms, you can’t collect data based on collection guidance without also knowing and including the evaluation guidance.  This makes any centralized database of security data more difficult to create and less efficiently implemented.

I suppose I would summarize by saying that I agree that OVAL, as it is now, *could* be used to accomplish the scenario (and many others), but could more optimally be implemented in a way that not only supports those but potentially other uses as well.

Best,
David

On May 7, 2015, at 9:20 AM, Hansbury, Matt <[email-masked]<mailto:[email-masked]>> wrote:

All,

I’d like to call your attention to the following Internet Draft that I submitted on Tuesday evening:

https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping/

This document discusses how the Open Vulnerability and Assessment Language (OVAL) can be leveraged to fulfill some aspects defined by the SACM Information Model.  Specifically, it discusses the various components of the Information Model where the OVAL Language could be used either fully or partially as a Data Model for those components.  Furthermore, it provides some lessons learned over the course of the past 10+ years by the OVAL community and some concrete recommendations on how best to utilize the different models of the OVAL Language within SACM.

We welcome your feedback and/or questions on the document and encourage you to review ahead of the virtual interim meeting next week, where we will spend a few minutes talking about the draft.

Thanks
Matt

============================================
Matthew Hansbury
J83D - Cyber Security Partnerships, Sharing, and Automation
The MITRE Corporation
Email: [email-masked]<mailto:[email-masked]>

_______________________________________________
sacm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/sacm

-David

--
David E. Ries
Partner
Farnam Hall Ventures

farnamhallventures.com<http://farnamhallventures.com>

---

From: David Ries <[email-masked]>
Date: Wed, 3 Jun 2015 13:04:08 -0500

Hi Matt,

Thanks for the feedback. This is really helpful. A few, brief comments below…

-David

> On Jun 3, 2015, at 9:39 AM, Hansbury, Matt <[email-masked]> wrote:
> 
> Hi David,
>  
> From: David Ries [mailto:[email-masked] <mailto:[email-masked]>] 
> Cc: [email-masked] <mailto:[email-masked]>

DER: Yes, but protocols are just one example. My point is that OVAL is really very flexible. It seems to me that it can be used to make assertions about “stuff” and that “stuff” can protocol-based, system characteristic based, or anything else at higher or lower levels of abstraction.

> Retrieve large numbers of OVAL definitions (i.e. ~25k vulnerability and inventory definitions from Mitre, the Cisco IOS advisory definitions from Cisco, etc.)
> Analyze the definitions applicable to each platform in order to identify all of the data that must be collected from a managed device in order to resolve all the variables, generate items, etc. and evaluate the applicable definitons
> Collect and store that data
> Refresh that data when required by the organizations requirements for data freshness
> Evaluate any or all of the definitions on a schedule completely disconnected from the collection schedule
>  
> MJH:  You are correct that there is nothing about OVAL that stops implementers from doing smart things during data collection, David.  We would certainly agree that OVAL, as it currently is, could be used to support such a scenario as you laid out.  However, because OVAL Objects (the ‘what to collect’ part) and OVAL States (the ‘what the data should look like’ part) are tightly coupled within the OVAL Definitions model, this scenario is not as easily supported as it could be.  Any data collected must be part of an OVAL Definition, along with a corresponding OVAL State, which means that you can’t proactively collect all security relevant configuration information, without the corresponding what it should look like part.

DER: States are not required. If you want to collect data, the Object must be part of a Definition including a Test, but there’s no requirement to include a State. If you can express configuration information as an Object, you can certainly collect it proactively without any States.

The Definition and Test are making assertions, but in this scenario—call it an “inventory” or “audit” or “data collection” type of Definition—the Definition and Test are actually making assertions about how to handle the collection (i.e. whether the collection is successful if the Object doesn’t exist).

>   In SACM terms, you can’t collect data based on collection guidance without also knowing and including the evaluation guidance.

DER: Again, I don’t see this limitation. Collection guidance must specify what data to collect. OVAL would express this with an inventory-like definition specifying Objects to collect. Evaluation guidance must make assertions about specific pieces of data. OVAL expresses these assertions as Definitions, Tests, States, etc. and the specific data as Objects. 

I feel like I’m missing something that’s obvious to others, but OVAL seems to address these requirements quite nicely to me.

I guess it would be a more elegant fit if we added a Definition class for “data collection” and documentation to make this use case more explicit. And, to make evaluation guidance more elegant, we could allow Objects to be external and simply referenced by id. These seem like minor adjustments to me.

>   This makes any centralized database of security data more difficult to create and less efficiently implemented. 
>  

> On May 7, 2015, at 9:20 AM, Hansbury, Matt <[email-masked] <mailto:[email-masked]>> wrote:
>  
> All,
>  
> https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping/ <https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping/>
>  
> Email: [email-masked] <mailto:[email-masked]>
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/sacm <https://www.ietf.org/mailman/listinfo/sacm>
>  
> -David
>  
> --
> David E. Ries
> Partner
> Farnam Hall Ventures 
>  
> farnamhallventures.com <http://farnamhallventures.com/>
>  

-David

--
David E. Ries
Partner
Farnam Hall Ventures 

farnamhallventures.com <http://farnamhallventures.com/>

---

From: "Hansbury, Matt" <[email-masked]>
Date: Thu, 4 Jun 2015 15:27:22 +0000

Hi Jerome,

Thanks for the thoughtful comments.  Addressed some of these inline:

Thanks
Matt

-----Original Message-----
From: Jerome Athias [mailto:[email-masked]] 
Sent: Friday, May 22, 2015 6:06 AM
To: Hansbury, Matt
Cc: [email-masked]
Subject: Re: [sacm] OVAL/SACM IM mapping document posted

Hi,

after first review. Few comments:

First thing first, Thank you! ...
Lessons learned and gaps will be useful

I do appreciate honesty of recommending not to use OVAL Results Model, and highlighting known identified limitations of the OVAL language.

I concur and appreciate the lesson learned regarding SMEs involvement.
(it is needed and critical while 'this thing' is just BIG, and could allow Domain-driven design [1], consensus-per-domain). And we should not limit innovation (Ref. 6.6.1)

KISS principle is good [2]. And I would add to this that it starts with a proper and intelligent Taxonomy (while manipulated concepts could seem obvious, it is not the case. e.g.: Event [6], system, endpoint, ...)

Following Gunnar comments and my modest point of view on the current OVAL System Characteristics Model (and few other MITRE models), I would be happy to share my comments/arguments for some enhancements
(e.g.: regarding the class types). [5]
(6.3.1 "it will always require a frequent revision cycle") (and that could help fighting against the "OVAL content is costly to create" ref. 6.9.1 statement)

Regarding the pure wording of the document, in case you decide to update it; 3.2.
I suggest adding the concept of controlled vocabularies close to the enumerations concept I suggest adding "reducing redundancy" + "and providing better interoperability and automation"

MJH:  We can certainly add something like this. 

4.1.
regarding "The OVAL Language does not provide any mechanism for instructing tools where to send collected data..."
why not introducing CAESARS? [3]

MJH:  CAESARS may well be a way that this can be accomplished. For the purposes of this paper, we focused on OVAL and how it relates to the SACM Information Model alone and tried to keep away from how other specific things might fit in.  I most certainly would like to discuss those type of topics over the email list, however.  

4.5.
"the OVAL Definitions Model could be modified to support monitoring of events."
Please kindly introduce this to STIX community asap! [4]

"The OVAL System Characteristics Model ... can be used to implement a data format for sharing collected data."
Please kindly come discuss this in the STIX community.

MJH:  I'm afraid I'm not sure how this relates to STIX.  Are you suggesting that STIX has solved this problem?

"a standardized format is required"
I could try to sell you some XORCISM / IVIL here [5]

4.3.
"time interval"
This is discussed in STIX community also

"event"
see https://cybox.mitre.org/about/faqs.html#B15
(Object/Observable and State...)

MJH:  Over the years we have discussed how and if CyBOX fits into OVAL, even thinking about whether CyBOX could be used to format the collected data (in place or augmenting the OVAL System Characteristics model). This is probably a topic for this group to discuss, since CyBOX is a schema for encoding cyber related information.  My understanding of CyBOX is supposed to be generic enough to support use cases as varying as ours, it is still being worked and might not be ready to do so at this time.  I would also note that we could, as a group, decide to support multiple data models, including both the OVAL System Characteristics Model and CyBOX (though probably a later version of it).  We could also decide to borrow things like event handling, etc. from CyBOX and support it that way.  Again, here in the context of this document, I am simply trying to identify a need for a solution, not the solution itself. 

6.3.2.
"SACM should focus on providing the foundational building blocks"
My lesson learned in addition: Taxonomy, definitions coming from the SMEs are needed.
+
6.4.2.
SMEs should define the objects and enumerations/vocabularies (while finding the appropriate "balance", ref. 6.6.2.)

6.5.2.
I suggest thinking about the concept of Maturity Levels (or Levels of
requirements)

MJH:  Can you help me understand how that would help with this lesson?  The referenced lesson to encourage vendor participation by providing an incentive to do so that resonates with them.  I'm not quite sure how introducing Maturity Levels would aid in implementing that lesson, but I'd be interested in hearing your thoughts.

6.7.2.
I do agree that Context is critical to capture.

6.8.1.
The concept of Confidence (as opposed to Fidelity) is important in XORCISM [5] and currently under discussion in the STIX community.

MJH:  Perhaps fidelity is the wrong word here.  I agree that confidence is important and we should be able (I think) to provide confidence for both results as well as the data collected itself.  When I say fidelity, I meant more along the lines of the amount of information returned as part of the results.  We recommend providing flexibility with respect to the amount of result information returned, since in some cases, you simply want *all* information, while in others you may only be interested in a subset or a rolled up result. 

4.2.
typo:
"security posture of one more endpoints"
should be
"security posture of one or more endpoints"

MJH:  Good catch, will fix.

[1] https://en.wikipedia.org/wiki/Domain-driven_design

[2] https://en.wikipedia.org/wiki/KISS_principle

[3] http://www.dhs.gov/continuous-asset-evaluation-situational-awareness-and-risk-scoring-reference-architecture-report

[4] http://stix.mitre.org/

[5] http://www.frhack.org/research/xorcism.php
https://www.cupfighter.net/2010/10/ivil-an-xml-schema-to-exchange-vulnerability-information

[6] https://cybox.mitre.org/about/faqs.html#B15

2015-05-07 17:20 GMT+03:00 Hansbury, Matt <[email-masked]>:
> This document discusses how the Open Vulnerability and Assessment 
> Language
> (OVAL) can be leveraged to fulfill some aspects defined by the SACM 
> Information Model.  Specifically, it discusses the various components 
> of the Information Model where the OVAL Language could be used either 
> fully or partially as a Data Model for those components.  Furthermore, 
> it provides some lessons learned over the course of the past 10+ years 
> by the OVAL community and some concrete recommendations on how best to 
> utilize the different models of the OVAL Language within SACM.
>
>
>
> We welcome your feedback and/or questions on the document and 
> encourage you to review ahead of the virtual interim meeting next 
> week, where we will spend a few minutes talking about the draft.
>
>
>
> Thanks
>
> Matt
>
>
>
>
>
> Email: [email-masked]
>
>
>
>
> [email-masked]

---

From: "Hansbury, Matt" <[email-masked]>
Date: Thu, 4 Jun 2015 15:36:27 +0000

Hi Jerome,

Do have some specifics on this?  The Lessons Learned were very focused on what we've learned while working with OVAL over the past 10+ years.  While we are quite aware of SWID, I'm not quite sure what you had in mind here.  

Thanks
Matt

-----Original Message-----
From: Jerome Athias [mailto:[email-masked]] 
Sent: Friday, May 22, 2015 12:44 PM
To: Hansbury, Matt
Cc: [email-masked]
Subject: Re: [sacm] OVAL/SACM IM mapping document posted

I would be interested, if possible, to have a feedback or lesson learnt regarding SWID.
Thanks