Subject: [scap_interest] Gaps in Risk Management

From: Adam Montville <[email-masked]>
Date: Tue, 14 Feb 2012 20:51:03 +0000

All,

Security automation seeks to automate what we can ultimately within the framework of risk management.  Scoring is not enough to measure risk.  What appears to be needed is a method by which we can automate risk measurement.  It would be interesting to discuss various ways in which such automation might be achieved along both qualitative and quantitative lines.

The scoring methods we have in place today for vulnerabilities and configurations (CVSS and CCSS respectively) can measure risk of a particular instance of a class of concepts in our domain.  What I don't see being possible at this point is a way to measure the aggregate instances of vulnerabilities and misconfigurations for some set of assets (whether that be a single asset or a composite asset representing an entire network).

Risk management methodologies are not scarce – NIST, ISO, CobiT, OCTAVE and others all present some method of managing risk, and they are similar.  How can we apply automation in support of these various risk management methods while providing both qualitative and quantitative risk assessment under the hood?

Regards,

Adam W. Montville | Security and Compliance Architect

Direct: 503 276-7661
Mobile: 360 471-7815

TRIPWIRE | Take CONTROL
http://www.tripwire.com

---

From: Karen Scarfone <[email-masked]>
Date: Tue, 14 Feb 2012 15:57:18 -0500

A few quick thoughts on this:

* We can't aggregate all these vulnerabilities unless we can model their
relationships. For example, a particular configuration setting might
nullify the applicability of a particular software flaw, and configuration
settings certainly affect each other.

* There's a whole other class of vulnerabilities besides software flaws and
configurations. Basically, any other vulnerability can be considered a
trust misuse/abuse vulnerability: social engineering and insider attacks,
for example. A risk management methodology that addresses software flaws
and misconfigurations is inadequate if it doesn't capture the other
vulnerabilities as well. We do have a spec, CMSS, designed to provide
scoring for these vulnerabilities just like CVSS does for software flaws
and CCSS does for misconfigurations. What we don't yet have is a (heaven
help me) dictionary or taxonomy of CMSS vulnerabilities. (Fortunately, I'd
expect this to be pretty small and straightforward to create, compared to
CVE and CCE).

Karen

On Tue, Feb 14, 2012 at 3:51 PM, Adam Montville <[email-masked]>wrote:

> All,
>
> Security automation seeks to automate what we can ultimately within the
> framework of risk management.  Scoring is not enough to measure risk.  What
> appears to be needed is a method by which we can automate risk measurement.
>  It would be interesting to discuss various ways in which such automation
> might be achieved along both qualitative and quantitative lines.
>
> The scoring methods we have in place today for vulnerabilities and
> configurations (CVSS and CCSS respectively) can measure risk of a
> particular instance of a class of concepts in our domain.  What I don't see
> being possible at this point is a way to measure the aggregate instances of
> vulnerabilities and misconfigurations for some set of assets (whether that
> be a single asset or a composite asset representing an entire network).
>
> Risk management methodologies are not scarce – NIST, ISO, CobiT, OCTAVE
> and others all present some method of managing risk, and they are similar.
>  How can we apply automation in support of these various risk management
> methods while providing both qualitative and quantitative risk assessment
> under the hood?
>
> Regards,
>
> _______________________________________________
> scap_interest mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scap_interest
>

-- 
Karen Scarfone, Principal Consultant, Scarfone Cybersecurity
[email-masked]   (703)401-1018

---

From: Adam Montville <[email-masked]>
Date: Tue, 14 Feb 2012 21:07:37 +0000

Great points, Karen!  These are issues that should be discussed.  Maybe the end goal would be to get to an ability to represent risk, but the interim step is to more completely represent vulnerabilities and their relationships.  Whatever we come up with for the automation piece should certainly feed into a larger risk management picture which could take into account the "human vulnerability" measurements.  In other words, I believe that we can provide risk measurement in a modular way, where conveying misconfiguration risk would complement vulnerability risk, and vice versa, without necessarily requiring the other elements of risk management to be present – these would feed up into the risk management system.

I could see vocabulary of relationships being established as part of closing these gaps.  At the same time, we could be working to understand the variety of risk management frameworks that are out there and determine how to feed our data into those in an appropriate manner – this seems to imply a risk reporting effort of some kind.

BTW, I wholeheartedly agree that relating configuration items to other configuration items is an issue that needs to be addressed.  If we look at many control frameworks, we'll see some control with a derived requirement for "strong authentication."  What does that mean for a Windows Server 2008 box managing users with user/pass credentials?  We have to take into account not just password length, history, complexity and the like, but also account management settings (lockout, threshold and so on) and attack effectiveness as well (so it's not just relating configuration items to other configuration items).  It becomes rather complicated.

Adam

From: Karen Scarfone <[email-masked]<mailto:[email-masked]>>
Date: Tue, 14 Feb 2012 15:57:18 -0500
To: Adam Montville <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [scap_interest] Gaps in Risk Management

* We can't aggregate all these vulnerabilities unless we can model their relationships. For example, a particular configuration setting might nullify the applicability of a particular software flaw, and configuration settings certainly affect each other.

* There's a whole other class of vulnerabilities besides software flaws and configurations. Basically, any other vulnerability can be considered a trust misuse/abuse vulnerability: social engineering and insider attacks, for example. A risk management methodology that addresses software flaws and misconfigurations is inadequate if it doesn't capture the other vulnerabilities as well. We do have a spec, CMSS, designed to provide scoring for these vulnerabilities just like CVSS does for software flaws and CCSS does for misconfigurations. What we don't yet have is a (heaven help me) dictionary or taxonomy of CMSS vulnerabilities. (Fortunately, I'd expect this to be pretty small and straightforward to create, compared to CVE and CCE).

Karen

On Tue, Feb 14, 2012 at 3:51 PM, Adam Montville <[email-masked]<mailto:[email-masked]>> wrote:
All,

Regards,

Direct: 503 276-7661
Mobile: 360 471-7815

TRIPWIRE | Take CONTROL
http://www.tripwire.com

_______________________________________________
scap_interest mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/scap_interest

--
Karen Scarfone, Principal Consultant, Scarfone Cybersecurity
[email-masked]<mailto:[email-masked]>   (703)401-1018

---

From: Jerome Athias <[email-masked]>
Date: Tue, 14 Feb 2012 22:34:10 +0000

Hi list,

could it be possible to have a brainstorming at the RSA Conference?

or / because i identifed the following talk (at BSides San Francisco, // 
free event):

*Metrics That Don’t Suck: A New Way To Measure Security Effectiveness ~ 
Dr. Mike Lloyd*

How does your organization measure and report its security posture and 
performance?  Do you have spreadsheets that show how many 
vulnerabilities you found last month, or how many viruses your AV system 
stopped? Those numbers might pacify your management, but any security 
pro can tell you that they are no way to benchmark the real work you do 
– or how much danger your enterprise might be in.

Maybe the problem is that we’re all trying to use the data we already 
have – host metrics, network metrics, applications data – instead of 
building the data we actually need.  We need metrics that show the 
current range of threats, and the enterprise’s exposure. We need data 
that shows whether our security tools and programs are actually working 
or not. We need methods for demonstrating that our security teams are 
performing well – not only this month, but over a period of time.

In this thought-provoking presentation, we’ll describe methods for 
building an enterprise security metrics program that’s completely 
different from the current, sucky model of counting vulnerabilities or 
numbers of patches applied. We’ll outline methods for monitoring the 
threat landscape, and your organization’s exposure. We’ll offer some 
best practices for measuring the effectiveness of current security tools 
and systems. Best of all, we’ll outline a way to build a maturity model 
for security, so that you can show your security team’s performance on a 
month-to-month basis, and demonstrate its continuing improvement over time.

Want to stop reporting a bunch of crap and start building a real set of 
data that accurately measures your organization’s risk and its 
effectiveness in controlling it?  Want to learn how to integrate 
security data across hosts, networks, and applications?  Want your 
performance – and your company’s security posture – to be monitored 
using metrics that don’t suck?  Here’s a chance to look at the picture 
from a whole new angle.

(ref: 
http://www.infosecisland.com/blogview/20057-Security-BSides-San-Francisco-Speakers-and-Topics-Lineup.html 
)

I see these events as a good place to exchange our ideas.

Just my 2 cents

/JA

---

From: <[email-masked]>
Date: Tue, 14 Feb 2012 17:47:51 -0500

Hi Jerome,

I think it is a really good idea to start from use cases and what we want to get out of this work to drive the standards efforts.  The work being done here should result in the ability to report on meaningful and actionable metrics.  Brainstorming like this at the RSA Conference and on list would be great.  Activities like this may be tied to the frameworks as well - how do we get good risk metrics against the frameworks in which we want to measure our security programs?

Luis brought up CSA's control matrix, maybe we want to think about what was done for 800-53 and how that could apply out to other frameworks like ISO27002 +ISO27017 (Virtualization/cloud), HIPAA HITECH, etc.  performing checks, assessing risk, and producing metrics are the drivers for these underlying activities.

Dan Greer has some good presentations online related to metrics and is running a series in IEE Security and Privacy on the topic as well.  I do need to admit that I have to catch up on my reading, so I can't tell you too much about that effort :)

Any other thoughts on these topics?

Thanks,
Kathleen

From: [email-masked] [mailto:[email-masked]] On Behalf Of Jerome Athias
Sent: Tuesday, February 14, 2012 5:34 PM
To: [email-masked]
Subject: Re: [scap_interest] Gaps in Risk Management

Hi list,

or / because i identifed the following talk (at BSides San Francisco, // free event):

Metrics That Don't Suck: A New Way To Measure Security Effectiveness ~ Dr. Mike Lloyd

How does your organization measure and report its security posture and performance?  Do you have spreadsheets that show how many vulnerabilities you found last month, or how many viruses your AV system stopped? Those numbers might pacify your management, but any security pro can tell you that they are no way to benchmark the real work you do - or how much danger your enterprise might be in.

Maybe the problem is that we're all trying to use the data we already have - host metrics, network metrics, applications data - instead of building the data we actually need.  We need metrics that show the current range of threats, and the enterprise's exposure. We need data that shows whether our security tools and programs are actually working or not. We need methods for demonstrating that our security teams are performing well - not only this month, but over a period of time.

In this thought-provoking presentation, we'll describe methods for building an enterprise security metrics program that's completely different from the current, sucky model of counting vulnerabilities or numbers of patches applied. We'll outline methods for monitoring the threat landscape, and your organization's exposure. We'll offer some best practices for measuring the effectiveness of current security tools and systems. Best of all, we'll outline a way to build a maturity model for security, so that you can show your security team's performance on a month-to-month basis, and demonstrate its continuing improvement over time.

Want to stop reporting a bunch of crap and start building a real set of data that accurately measures your organization's risk and its effectiveness in controlling it?  Want to learn how to integrate security data across hosts, networks, and applications?  Want your performance - and your company's security posture - to be monitored using metrics that don't suck?  Here's a chance to look at the picture from a whole new angle.
(ref: http://www.infosecisland.com/blogview/20057-Security-BSides-San-Francisco-Speakers-and-Topics-Lineup.html )

Just my 2 cents

/JA

---

From: Luis Nunez <[email-masked]>
Date: Tue, 14 Feb 2012 17:59:26 -0500

Jerome,
great idea and thanks for mentioning this talk. It would be an interesting discussion to have with Dr. Mike on the topic.  Lets hook up.

Since you mentioned Geer.  Here is an excellent talk Dan recent gave. http://geer.tinho.net/geer.suitsandspooks.8ii12.txt

-ln

On Feb 14, 2012, at 5:47 PM, <[email-masked]> wrote:

> Hi Jerome,
>  
> I think it is a really good idea to start from use cases and what we want to get out of this work to drive the standards efforts.  The work being done here should result in the ability to report on meaningful and actionable metrics.  Brainstorming like this at the RSA Conference and on list would be great.  Activities like this may be tied to the frameworks as well – how do we get good risk metrics against the frameworks in which we want to measure our security programs? 
>  
> Luis brought up CSA’s control matrix, maybe we want to think about what was done for 800-53 and how that could apply out to other frameworks like ISO27002 +ISO27017 (Virtualization/cloud), HIPAA HITECH, etc.  performing checks, assessing risk, and producing metrics are the drivers for these underlying activities.
>  
> Dan Greer has some good presentations online related to metrics and is running a series in IEE Security and Privacy on the topic as well.  I do need to admit that I have to catch up on my reading, so I can’t tell you too much about that effort J
>  
> Metrics That Don’t Suck: A New Way To Measure Security Effectiveness ~ Dr. Mike Lloyd
> 
> How does your organization measure and report its security posture and performance?  Do you have spreadsheets that show how many vulnerabilities you found last month, or how many viruses your AV system stopped? Those numbers might pacify your management, but any security pro can tell you that they are no way to benchmark the real work you do – or how much danger your enterprise might be in.
> 
> Maybe the problem is that we’re all trying to use the data we already have – host metrics, network metrics, applications data – instead of building the data we actually need.  We need metrics that show the current range of threats, and the enterprise’s exposure. We need data that shows whether our security tools and programs are actually working or not. We need methods for demonstrating that our security teams are performing well – not only this month, but over a period of time.
> 
> In this thought-provoking presentation, we’ll describe methods for building an enterprise security metrics program that’s completely different from the current, sucky model of counting vulnerabilities or numbers of patches applied. We’ll outline methods for monitoring the threat landscape, and your organization’s exposure. We’ll offer some best practices for measuring the effectiveness of current security tools and systems. Best of all, we’ll outline a way to build a maturity model for security, so that you can show your security team’s performance on a month-to-month basis, and demonstrate its continuing improvement over time.
> 
> Want to stop reporting a bunch of crap and start building a real set of data that accurately measures your organization’s risk and its effectiveness in controlling it?  Want to learn how to integrate security data across hosts, networks, and applications?  Want your performance – and your company’s security posture – to be monitored using metrics that don’t suck?  Here’s a chance to look at the picture from a whole new angle.
>

---

From: Adam Montville <[email-masked]>
Date: Wed, 15 Feb 2012 13:57:45 +0000

Brainstorming at RSA is an excellent idea.  I'd like to additionally look at what was done with risk external to the information security domain.

From: Luis Nunez <[email-masked]<mailto:[email-masked]>>
Date: Tue, 14 Feb 2012 17:59:26 -0500
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Mike Lloyd <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>
Subject: Re: [scap_interest] Gaps in Risk Management

Jerome,

-ln

On Feb 14, 2012, at 5:47 PM, <[email-masked]<mailto:[email-masked]>> wrote:

Hi Jerome,

Dan Greer has some good presentations online related to metrics and is running a series in IEE Security and Privacy on the topic as well.  I do need to admit that I have to catch up on my reading, so I can’t tell you too much about that effort :)

Thanks,
Kathleen

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Jerome Athias
Sent: Tuesday, February 14, 2012 5:34 PM
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [scap_interest] Gaps in Risk Management

Hi list,

Just my 2 cents

/JA

_______________________________________________ scap_interest mailing list [email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/scap_interest

---

From: "Lloyd, Mike" <[email-masked]>
Date: Wed, 15 Feb 2012 07:06:56 -0800 (PST)

(Hopefully this is not a duplicate for most folks - I wrote a note below
while not yet a list member, but have since joined & cancelled my earlier
attempt to send.  I would be most interested in SCAP and metrics related
discussion with anyone attending either BSides or RSA in San Francisco
shortly.)

Mike

From: Lloyd, Mike [mailto:[email-masked]] 
Sent: Tuesday, February 14, 2012 3:36 PM
To: 'Luis Nunez'; [email-masked]'
Cc: [email-masked]'; [email-masked]'
Subject: RE: [scap_interest] Gaps in Risk Management

Absolutely - thanks for adding me to the thread, Luis.  As Jerome noted,
I'm speaking at BSides - an event I've found can be quite worthwhile.
I'll also be speaking on the show floor at RSA (at the RedSeal Networks
booth), but the show floor talk will necessarily be much less detailed.
I'd be happy to discuss metrics with members of this group at either of
these venues, or online.

Some recent comments I made on metrics showed up here:

http://www.darkreading.com/security-monitoring/167901086/security/perimete
r-security/232600679/five-tactical-security-metrics-to-watch.html

For folks thinking about SCAP, I didn't highlight SCAP in that interview,
but I do see SCAP as a fundamental building block of the metrics I
recommend.  Specifically, correlation of CVE instances and correlation
with the CVSS framework are fundamental.  However, I find I have to build
a few layers on top of that - at first into the "environmental" component,
but eventually beyond that, because I need to generate not just metrics on
vulnerabilities, but also on hosts, and even for an entire network.

Anyone who has a BrightTalk login, or is willing to register for one, can
also review a recent webinar we recorded - a co-presentation by Mike
Rothman of Securosis (who covers general themes and difficulties) and
myself (covering our actual measurement approach, in a short segment at
the end):

http://www.brighttalk.com/webcast/7245/39267

Regards

Mike

From: Luis Nunez [mailto:[email-masked]] 
Sent: Tuesday, February 14, 2012 2:59 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; Mike Lloyd
Subject: Re: [scap_interest] Gaps in Risk Management

Jerome,

great idea and thanks for mentioning this talk. It would be an interesting
discussion to have with Dr. Mike on the topic.  Lets hook up.

Since you mentioned Geer.  Here is an excellent talk Dan recent gave.
http://geer.tinho.net/geer.suitsandspooks.8ii12.txt

-ln

Hi Jerome,

I think it is a really good idea to start from use cases and what we want
to get out of this work to drive the standards efforts.  The work being
done here should result in the ability to report on meaningful and
actionable metrics.  Brainstorming like this at the RSA Conference and on
list would be great.  Activities like this may be tied to the frameworks
as well - how do we get good risk metrics against the frameworks in which
we want to measure our security programs? 

Luis brought up CSA's control matrix, maybe we want to think about what
was done for 800-53 and how that could apply out to other frameworks like
ISO27002 +ISO27017 (Virtualization/cloud), HIPAA HITECH, etc.  performing
checks, assessing risk, and producing metrics are the drivers for these
underlying activities.

Dan Greer has some good presentations online related to metrics and is
running a series in IEE Security and Privacy on the topic as well.  I do
need to admit that I have to catch up on my reading, so I can't tell you
too much about that effort J

Thanks,

Kathleen

From: [email-masked]
[mailto:[email-masked]] On Behalf Of Jerome Athias
Sent: Tuesday, February 14, 2012 5:34 PM
To: [email-masked]
Subject: Re: [scap_interest] Gaps in Risk Management

Hi list,

Metrics That Don't Suck: A New Way To Measure Security Effectiveness ~ Dr.
Mike Lloyd

How does your organization measure and report its security posture and
performance?  Do you have spreadsheets that show how many vulnerabilities
you found last month, or how many viruses your AV system stopped? Those
numbers might pacify your management, but any security pro can tell you
that they are no way to benchmark the real work you do - or how much
danger your enterprise might be in.

Maybe the problem is that we're all trying to use the data we already have
- host metrics, network metrics, applications data - instead of building
the data we actually need.  We need metrics that show the current range of
threats, and the enterprise's exposure. We need data that shows whether
our security tools and programs are actually working or not. We need
methods for demonstrating that our security teams are performing well -
not only this month, but over a period of time.

In this thought-provoking presentation, we'll describe methods for
building an enterprise security metrics program that's completely
different from the current, sucky model of counting vulnerabilities or
numbers of patches applied. We'll outline methods for monitoring the
threat landscape, and your organization's exposure. We'll offer some best
practices for measuring the effectiveness of current security tools and
systems. Best of all, we'll outline a way to build a maturity model for
security, so that you can show your security team's performance on a
month-to-month basis, and demonstrate its continuing improvement over
time.

Want to stop reporting a bunch of crap and start building a real set of
data that accurately measures your organization's risk and its
effectiveness in controlling it?  Want to learn how to integrate security
data across hosts, networks, and applications?  Want your performance -
and your company's security posture - to be monitored using metrics that
don't suck?  Here's a chance to look at the picture from a whole new
angle.

(ref:
http://www.infosecisland.com/blogview/20057-Security-BSides-San-Francisco-
Speakers-and-Topics-Lineup.html )

Just my 2 cents

/JA