Subject: [mile] Redesign of indicators

From: "Roman D. Danyliw" <[email-masked]>
Date: Sat, 1 Mar 2014 06:49:44 +0000

Hello!

Nearly a year ago, the challenge with adding additional indicator sharing classes to the existing incident reporting data model in IODEF was noted [1, 
http://www.ietf.org/mail-archive/web/mile/current/msg00985.html].  As we enumerate the outstanding actions to bring 5070bis to closure, this central issue needs to be resolved.  The text is littered with references to data elements representing some part of "the incident" making it awkward for an implementer solely interested in sharing indicators.  However, for all of the interest in using IODEF to convey indicators,  two key constructs added 
to the data model to support indicators (i.e., @indicator-uid/set-id and attributes with "-watchlist" values) remain undocumented (see Issue #41 and #42).  Furthermore, Issue #14, to add predicate logic to these indicators is unresolved.

This email is a _rough_ proposal to bridge these two use cases.  The summary of this proposal is that we should think of indicators as being the product of analyzing an incident.  Sometimes, an IODEF document will only describe the specific events an incident; it might describe the incident and the indicators derived from it; or in another instance the details of the incident might not be shared at all but the derived indicators would be.  

This proposal suggests that we move all indicator-specific information to a new distinct class, IndicatorData, that is a child of Incident.  Indicators can be described outright in this new class, or can be defined by reference by pointing to specific data (e.g., Address or DomainData)  already described elsewhere in the IODEF document.  This proposal addresses current open issues #14 (predicate logic for indicators) and #28 (specifying time windows for indicators) by suggesting a data model to cover these use cases.  Issue #41 (documenting @indicator-* attributes) is resolved by redefining @indicator-uid and removing @indicator-set-id.  Issue #42 (documenting "-watchlist" attributes) is resolved by removing all these attribute values as they are no longer necessary.

Consider the following XML-snippets to get a sense of how such an approach would look:

-------------[ Begin XML Examples ]-----------------
<EventData>
...
   <System>
     <Node>
        <DomainData observable-id="observable-1">
          <Name>www4.example.com</Name>
        </DomainData>
     </Node>
   </System>

</EventData>
...
<IndicatorData>
  <!-- Indicator #1 (a watchlist) -->
  <!-- set of domain names as indicators -->
  <!--   shows an IndicatorExpression -->
  <Indicator>
      <IndicatorID version="1.0">indicator-1</IndicatorID>
      <Description>My First Indicator</Description>
      <IndicatorExpression operator="or">
         <Observable>
           <DomainData><Name>www1.example.com</Name></DomainData>
         </Observable>
         <Observable>
           <DomainData><Name>www2.example.com</Name></DomainData>
         </Observable>
         <Observable>
           <DomainData><Name>www3.example.com</Name></DomainData>
         </Observable>
      </IndicatorExpression>
  </Indicator>

  <!-- Indicator #2 -->
  <!-- just one domain name -->
  <Indicator>
      <IndicatorID version="1.0">indicator-2</IndicatorID>
      <Description>My Second Indicator</Description>
      <Observable>
         <DomainData><Name>www1.example.com</Name></DomainData>
      </Observable>
  </Indicator>

  <!-- Indicator #3 -->
  <!-- just one domain name -->
  <Indicator>
      <IndicatorID version="1.0">indicator-3</IndicatorID>
      <Description>My Third Indicator</Description>
      <Observable>
         <DomainData><Name>www1.example.com</Name></DomainData>
      </Observable>
  </Indicator>

  <!-- Indicator #4 -->
  <!-- just one domain name -->
  <Indicator>
      <IndicatorID version="1.0">indicator-4</IndicatorID>
      <Description>My Fourth Indicator</Description>
      <Observable>
         <DomainData><Name>www1.example.com</Name></DomainData>
      </Observable>
  </Indicator>

  <!-- Indicator #5 -->
  <!-- equivalent to Indicator #1 but uses references to #2,3,4 -->
  <!--   shows an IndicatorReference -->
  <Indicator>
      <IndicatorID version="1.0">indicator-5</IndicatorID>
      <Description>My Fifth Indicator</Description>
      <IndicatorExpression operator="or">
         <IndicatorReference uid-ref="indicator-2" />
         <IndicatorReference uid-ref="indicator-3" />
         <IndicatorReference uid-ref="indicator-4" />
      </IndicatorExpression>
  </Indicator>

  <!-- Indicator #6 -->
  <!-- pulls in an observable from EventData -->
  <!--    shows an ObservableReference -->
  <!--    shows the valid time interval -->
  <Indicator>
      <IndicatorID version="1.0">indicator-6</IndicatorID>
      <Description>My Sixth Indicator</Description>
      <StartTime>2013-06-08T05:44:53-05:00</StartTime>
      <EndTime>2014-06-08T05:44:53-05:00</EndTime>
      <ObservableReference uid-ref="observable-1" />
  </Indicator>

  <!-- Indicator #7 -->
  <!-- pulls in an external indicator not in this document -->
  <!--    shows an IndicatorReference@euid-ref -->
  <Indicator>
      <IndicatorID version="1.0">indicator-7</IndicatorID>
      <Description>My Seventh Indicator</Description>
      <IndicatorReference euid-ref="certcc-indicator-2390823094-23423" />
  </Indicator>

  <!-- Indicator #8 -->
  <!-- shows the expression: "(indicator-2 or indicator-3) AND indicator-4"
  <Indicator>
      <IndicatorID version="1.0">indicator-8</IndicatorID>
      <Description>My Eight Indicator</Description>
      <IndicatorExpression operator="and">
         <IndicatorExpression operator="or">
           <IndicatorReference uid-ref="indicator-2" />
           <IndicatorReference uid-ref="indicator-3" />
         </IndicatorExpression>
         <IndicatorReference uid-ref="indicator-4" />
      </IndicatorExpression>
  </Indicator>
</IndicatorData>
-------------[ End XML Examples ]-----------------

The following is rough text that describes the data model:

-------------[ Begin Draft Text ]-----------------

3.28.  IndicatorData Class

   The IndicatorData class describes the indicators identified from
   analysis of an incident.

   +--------------------------+
   | IndicatorData            |
   +--------------------------+
   |                          |<>--{1..*}--[ Indicator      ]
   +--------------------------+

   The aggregate class that constitutes IndicatorData is:

   Indicator 
      One or more.  An indicator from the incident.

   The IndicatorData class has no attributes.

3.29.  Indicator Class

   The Indicator class describes a cyber indicator.  An indicator
   consists of observable features and phenomenon that aid in the
   forensic or proactive detection of malicious activity, and associated
   meta-data.  This indicator can be described outright or reference
   observable features and phenomenon described elsewhere in the
   incident information.  Portions of an incident description can be
   composed to define an indicator, as can the indicators themselves.

   +--------------------+
   | Indicator          |
   +--------------------+
   | ENUM restriction   |<>----------[ IndicatorID            ]
   |                    |<>--{0..1}--[ AlternativeIndicatorID ]
   |                    |<>--{0..*}--[ Description            ]
   |                    |<>--{0..1}--[ StartTime              ]   |                    |<>--{0..1}--[ EndTime                ]   |                    

|<>--{0..1}--[ Confidence             ]
   |                    |<>--{0..*}--[ Contact                ]
   |                    |<>--{0..1}--[ Observable             ]
   |                    |<>--{0..1}--[ IndicatorExpression    ]
   |                    |<>--{0..1}--[ ObservableReference    ]
   |                    |<>--{0..1}--[ IndicatorReference     ]
   |                    |<>--{0..*}--[ AdditionalData         ]
   +--------------------+

   The aggregate classes that constitute Indicator are:

   IndicatorID 
      One.  An identifier for this indicator.  See Section 3.29.1

   AlternativeIndicatorID 
      Zero or one.  TODO.  See Section 3.29.2

   Description 
      Zero or more.  ML_STRING.  A free-form textual description of the
      indicator.

   StartTime       Zero or one.  DATETIME.  A timestamp of the start of the time      period during which this indicator is valid.   EndTime       

Zero or one.  DATETIME. A timestamp fo the end of the time period      during which this indicaor is valid.
   Confidence 
      Zero or one.  An estimate of the confidence in the quality of the
      indicator.  See Section 3.14.5.

   Contact 
      Zero or more.  Contact information for this indicator.  See
      Section 3.10.

   Observable 
      Zero or one.  An observable feature or phenomenon of this
      indicator.  See Section 3.29.3.

   IndicatorExpression 
      Zero or one.  A composition of observables.  See Section 3.29.4.

   ObservableReference 
      Zero or one.  A reference to a feature or phenomenon defined
      elsewhere in the document.  See Section 3.29.5.

   IndicatorReference 
      Zero or one.  A reference to an indicator.

   AdditionalData 
      Zero or more.  Mechanism by which to extend the data model.  See
      Section 3.9

   The Indicator class MUST have exactly one instance of an Observable,
   IndicatorExpression, ObservableReference, or IndicatorReference
   class.

   The StartTime and EndTime classes can be used to define an interval   during which the indicator is valid.  If both classes are present,   the 

indicator is consider valid only during the decribed interval.   If neither class is provided, the indicator is considered valid   during any time 

interval.  If only a StartTime is provided, the   indicator is valid anytime after this timestamp.  If only an EndTime   is provided, the indicator is 

valid anytime prior to this timestamp.
   The Indicator class has one attribute:

   restriction 
      Optional.  ENUM.  See Section 3.3.1.

3.29.1.  IndicatorID Class

   The IndicatorID class uniquely identifies an indicator.  ... TODO
   Discuss if (1) a CSIRT needs to keep state to make this globally
   unique like an IncidentID.  (2) An indicator generated by a given
   CSIRT MUST NOT reuse the same value ...

   The body of this class describes the unique identifier for an
   indicator (... TODO ... an "ID" type is not defined but this is
   equivalent to xs:ID).  An indicator is unique identified by the
   identifier in the body and the associated value of the version
   attribute.

   +------------------+
   | IndicatorID      |
   +------------------+
   | ID               |
   |                  |
   | STRING version   |
   +------------------+

   The IndicatorID class has three attributes:

   version 
      Required.  STRING.  A version number of an indicator.

3.29.2.  AlternativeIndicatorID Class

   The AlternativeIndicatorID class lists alternative identifiers for an
   indicator.

   +-------------------------+
   | AlternativeIndicatorID  |
   +-------------------------+
   | ENUM restriction        |<>--{1..*}--[ IndicatorReference ]
   |                         |
   +-------------------------+

   The aggregate class that constitutes AlternativeIndicatorID is:

   IndicatorReference 
      One or more.  A reference to an indicator.

   The AlternativeIndicatorID class has one attribute:

   restriction 
      Optional.  ENUM.  This attribute has been defined in Section 3.2.

3.29.3.  Observable Class

   The Observable class describes a feature and phenomenon that can be
   observed or measured for the purposes of detecting malicious
   behavior.

   +--------------------+
   | Observable         |
   +--------------------+
   |                    |<>--{0..1}--[ Address                     ]
   |                    |<>--{0..1}--[ DomainData                  ]
   |                    |<>--{0..1}--[ Service                     ]
   |                    |<>--{0..*}--[ EmailData                   ]
   |                    |<>--{0..1}--[ ApplicationHeader           ]
   |                    |<>--{0..*}--[ WindowsRegistryKeysModified ]
   |                    |<>--{0..1}--[ HashData                    ]
   |                    |<>--{0..1}--[ RecordData                  ]
   |                    |<>--{0..1}--[ Expectation                 ]
   |                    |<>--{0..1}--[ EventData                   ]
   |                    |<>--{0..1}--[ Incident                    ]
   |                    |<>--{0..1}--[ Reference                   ]
   |                    |<>--{0..1}--[ Assessment                  ]
   |                    |<>--{0..1}--[ HistoryItem                 ]
   |                    |<>--{0..*}--[ AdditionalData              ]
   +--------------------+

   The aggregate classes that constitute Observable are: ... TODO check
   the figure above while we discuss which classes to add here.  The
   figure lists every class that currently has an @indicator-uid.

   The Observable class MUST have exactly one of the possible child
   classes.

   The Observable class has no attributes.

3.29.4.  IndicatorExpression Class

   The IndicatorExpression describes an expression composed of observed
   phenomenon or features, or indicators.  Elements of the expression
   can be described directly, reference relevant data from other parts
   of a given IODEF document, or reference previously defined
   indicators.

   All child classes of a given instance of IndicatorExpression form a
   boolean algebraic expression where the operator between them is
   determined by the operator attribute.  Nesting an IndicatorExpression
   in itself is akin to a parenthesis in the expression.

   +--------------------------+
   | IndicatorExpression      |
   +--------------------------+
   | ENUM operator            |<>--{0..*}--[ IndicatorExpression  ]
   |                          |<>--{0..*}--[ Observable           ]
   |                          |<>--{0..*}--[ ObservableReference  ]
   |                          |<>--{0..*}--[ IndicatorReference   ]
   |                          |<>--{0..*}--[ AdditionalData       ]
   +--------------------------+

   The aggregate classes that constitute IndicatorExpression are:

   IndicatorExpression 
      Zero or more.  TODO.

   Observable 
      Zero or more.  TODO.

   ObservableReference 
      Zero or more.  TODO.

   IndicatorReference 
      Zero or more.  TODO.

   AdditionalData 
      Zero or more.  Mechanism by which to extend the data model.  See
      Section 3.9

   The IndicatorExpression class MUST have exactly one of the possible
   child classes.  ... TODO Additional text is required to describe the
   valid combinations of classes ...

   The IndicatorExpression class has one attributes:

   operator 
      Optional.  ENUM.  The operator.  TODO.

      1.  not.  negation operator.

      2.  and.  conjunction operator.

      3.  or.  disjunction operator.

3.29.5.  ObservableReference Class

   The ObservableReference describes a reference to an observable
   feature or phenomenon described elsewhere in the document.

   This class has no content.  ... TODO, @observable-id-ref uses an
   IDREF type, this is not yet defined in Section 2.  In schema it would
   be xs:IDREF .  We could also use xs:key ...

   +-------------------------+
   | ObservableReference     |
   +-------------------------+
   | EMPTY                   |
   |                         |
   | IDREF uid-ref           |
   +-------------------------+

   The ObservableReference class has one attributes:

   uid-ref 
      Required.  IDREF.  An identifier that serves as a reference to a
      class in the IODEF document.  The referenced class will have this
      identifier set in the observable-id attribute (TODO, NOTE:
      currently this attribute is named @indicator-uid).

3.29.6.  IndicatorReference Class

   The IndicatorReference describes a reference to an indicator.  This
   reference may be to an indicator described in the IODEF document or
   in a previously exchanged IODEF document.

   +--------------------------+
   | IndicatorReference       |
   +--------------------------+
   | EMPTY                    |
   |                          |
   | IDREF uid-ref            |
   | STRING euid-ref          |
   | STRING version           |
   +--------------------------+

   The IndicatorReference class has one attributes:

   uid-ref 
      Optional.  IDREF.  An identifier that serves as a reference to an
      Indicator class in the IODEF document.  The referenced Indicator
      class will have this identifier set in the IndicatorID class.

   euid-ref 
      Optional.  STRING.  An identifier that references an IndicatorID
      not in this IODEF document.

   version 
      Optional.  STRING.  A version number of an indicator.

   Either the uid-ref or the euid-ref attribute MUST be set.

-------------[ End Draft Text ]-----------------

Roman