Subject: RE: Comment's on Cynthia McLain's proposals

From: Oliver Dain <[email-masked]>
Date: Fri, 26 Oct 2001 17:47:14 -0400

Cynthia McLain asked me to respond to your concerns about including a
confidence measure in IDMEF.  My comments are below.  Please feel free to
respond to me with any further questions or comments you might have.

It is true that a confidence of .8 produced by one sensor is not the same
thing as a confidence of .8 on another sensor.  However, I don't think this
means that these numbers can't be useful.  In all cases the numbers reflect
the posterior probability of an attack given the data analyzed and the model
the sensor uses to do its analysis.  The model used and the data analyzed
will differ from sensor to sensor but that doesn't make the numbers useless.
There are many ways the values can be handled by the manager.  If we have a
rough idea of the effectiveness of the individual sensors in our network
(perhaps on different classes of attacks) it is then easy to combine the
probability estimates from several sensors as is commonly done when building
ensembles of classifiers in standard machine learning applications.  These
types of ensembles have been shown to improve classifier performance in a
number of domains and there is no reason to think it wouldn't work here as
well.  It is admittedly a bit tricky to figure out which alerts refer to the
same event, but I believe this type of information could be useful.

The system I am working on allows the user to write rules to specify the
security risk of various sequences of alerts.  The confidence measure
produced by various sensors would be very helpful in these applications as
well.

Many detectors produce probabilities.  For example, Emerald, our own DoS and
Probe detectors, the anomaly detectors written by Stephanie Forest et al.,
etc.  Even if the manager does nothing fancy with this confidence measures,
they could still be useful to an analyst.    A DoS alert from one of these
sensors with a confidence of .2 is not as serious as DoS alert with a
confidence of .9.  I don't think we should just discard this valuable
information because we are worried that some managers may not know how to
handle it.  They are free to ignore any information they believe to be
spurious.

> -------- Original Message --------
>
> Subject: Comment's on Cynthia McLain's proposals
> Date:    Fri, 26 Oct 2001 10:53:14 -0400
> From:    "David A. Curry" <[email-masked]>
> To:      Cynthia McLain <[email-masked]>
>
> ========================================
> Comments on "Confidence" (Message 4 of 4)
> =========================================
>
>   Cynthia> The confidence attribute was removed from the Alert data
> model due
>   Cynthia> to concerns within the IDWG over the ability to assign a
> meaningful
>   Cynthia> value to this field.
>
> Comment: I think the problem was a little more complex than that.  The
> issue
>   was no so much whether or not a meaningful value could be assigned to
> this
>   attribute by a single IDS.  Rather, the issue was what happens when
> you get
>   these values from different IDSes back at the manager -- are they
> comparable
>   in any meaningful way?
>
>   For example, if IDS-A says 0.7 and IDS-B says 0.8, does that mean B is
> more
>   certain than A?  Not necessarily, depending on how the number was
> calculated.
>   This becomes a real problem if both IDSes are reporting on the same
> event.
>   Say, for example, that a network-based IDS reports an event with
> confidence
>   0.8, and a host-based IDS reports the same event with a 0.2.  Does
> that mean
>   the network-based IDS is pretty sure it happened, and the host-based
> IDS is
>   pretty sure it didn't?  Not necessarily, since the confidences are not
>   directly comparable.  So which IDS does the manager believe?
>
>   That's the problem we couldn't solve.  I think.
>
> I'm not an expert on confidence calculations, and will defer to those
> who are
> for whether or not this attribute can be kept in a useful form.  I'm
> inclined
> not to include it in the IDMEF unless and until that can be shown.
>
> --Dave
>

---

From: "David A. Curry" <[email-masked]>
Date: Sat, 27 Oct 2001 10:50:25 -0400

Oliver's argument for "confidence" seems reasonable to me.  I would agree
with Cynthia's proposal that maybe we should bring it back as an optional
attribute.  If I recall, much of the original objection was that it was
mandatory, anyway.

Comments from the group?

As I recall, there were also some issues about what range it should have
(0.0 to 1.0, or 1 to 10, etc.) and what precision (0.x, 0.xx, 0.xxx,
etc.).  Those will come up again, so we would need to work them out as
part of including this, if we do include it.

--Dave

---

From: "David A. Curry" <[email-masked]>
Date: Sat, 27 Oct 2001 10:55:04 -0400

[My mailer hiccupped; apologies if this is a duplicate]

Comments from the group?

--Dave

---

From: Chris Lonvick <[email-masked]>
Date: Sat, 27 Oct 2001 22:15:24 -0500

>part of including this, if we do include it.

Hi Dave and All,

I still disagree with the concept.  As I see it, the purpose of 
producing an Internet standard, beyond just interoperability, is
for uniformity and consistency.  I have no doubt that individual
implementations would be able to give some credible value for
"confidence" but I would doubt that heterogeneous implementations
would give consistent or uniform values without explicit guidance.
For example, two analyzers may be placed in the same network 
path.  Both of them see the same traffic.  For a given set of 
circumstances, the first may send along an alert saying that it
saw Attack XYZ with a 80% confidence level.  The second may 
send along an alert giving only 20% confidence.  Without knowing
intimate details of the configurations and implementations of
the two analyzers, the values may confuse an administrator or an
automated parser.  Should it be believed because one analyzer
gives it a high confidence; should it be disbelieved because
they diverge; or should you just add the confidence values
together and divide them by 17 (my lucky number) to produce a
value I believe?  Should I toss out all analyzers that don't
give 100% confidence values when they send an alert?

The way to rectify the values would be to give guidance.  
Unfortunately, I think that this would have to be very explicit 
for each signature by defining the specific set of circumstances 
to give ranges of "confidence".  I would think that such an 
undertaking would require a large amount of time and effort.  If 
undertaken, I would suggest that it become a subsequent 
evolutionary addition to the IDMEF rather than a defined attribute
at this time.  

Thanks,
Chris