Subject: proposed IDMEF modifications (Message 2 of 4: host-based IDS)

From: Cynthia McLain <[email-masked]>
Date: Wed, 24 Oct 2001 08:39:05 -0400

1. Host-Based Intrusion Detection Systems
==========================================

The IDMEF specification captures the information generated by host-based
intrusion detection systems when they are alerting on process activity.
However, some host-based intrusion detection systems, such as TripWire
and BIDS (a program developed here at MIT Lincoln Laboratory), also
alert based on activities observed on the disk, such as file creations
and file modifications. Within the current IDMEF, it is necessary to
use Process and/or AdditionalData to represent this information.
Unfortunately, the lack of a predefined field to accommodate file
information results in vendor-specific representations for standard
information thereby defeating a stated objective of IDMEF.

In order to address this concern, the following changes are proposed
to better accommodate host-based intrusion detection systems and their
disk monitoring capabilities.

In addition to the issues mentioned above, we also noted some difficulty
defining messages because of the unclear semantics of "Source" and
"Target"
when the activity of interest is on a single computer. There was a brief
discussion on the IDWG mailing list in February 2001 about the
representation
of three relationships: Source/Destination, Client/Server, and
Attacker/Target as a single relationship: Source/Target. This topic
probably warrants more scrutiny but it is not addressed here beyond this
mention.

A. Alert impact types:
----------------------

The most recent version of IDMEF changes the "impact" attribute from an
enumerated type to a free text string. However, removing all structure
from the attribute will make it more difficult for Managers to integrate
alert information from different sources. Integration could be made more
difficult since the values, and even format, of the attribute will be
vendor-specific. If it is not feasible to return to the use of
enumerated
types, due to concerns about extensibility, the potential variability
in this attribute could be mitigated by recommending specific strings
in the document.

Should the previous enumerated types for the "impact" attribute be
reinstated or text recommended, additional values for this attribute
are desirable when the alerts are generated by host-based IDSs. Note
that in IDMEF v0.3 when an alert involves protected files, the value
of "impact" can only be "bad-unknown" or "unknown". To capture the
impact of file-based alerts, the following additional "impact" types
are proposed:

   Keyword        Description
   -------        -----------
   file-modified  A file was modified in some way. The modification may
be
                  to the name, content, ownership, and/or permissions.
   file-created   A file was created. The file may have been created in
a
                  protected directory, created with certain permission
(e.g.,
                  setuid), or created as a symbolic link to a protected
file
                  (as is done in exploiting a race condition).
   file-deleted   A protected file was deleted.
   file-read      A protected file was read.

B. Additional Node category type:
---------------------------------

In some cases, there may be a network of hosts that are not using
a network name resolver. In that case, on Unix systems, the names
may be resolved using the /etc/hosts file. In the current IDMEF
definition
it appears that in that case "unknown" would be used. In order to
eliminate
the use of "unknown" in that situation, the addition of a Node
"category"
type is proposed.

   Keyword        Description
   -------        -----------
   hosts          Local hosts file (Unix file system)

C. Additional Analyzer categories
---------------------------------

For a host-based intrusion detection system, the context in
which the files are being viewed becomes an important piece
of information. For that reason, two additional attributes
are proposed for the "Analyzer" element. Both attributes
are optional and are represented using free-form text. The
strings used should be the standard representation of that
information as reported by the OS.

The new XML representation is as follows:

<!ATTLIST Analyzer                 (
    analyzerid     CDATA      '0'
    manufacturer   CDATA      #IMPLIED
    model          CDATA      #IMPLIED
    version        CDATA      #IMPLIED
    class          CDATA      #IMPLIED
    ostype         CDATA      #IMPLIED
    osversion      CDATA      #IMPLIED
    %attlist.global;
)>

The two additional attributes are:

ostype
   Optional. Operating system name (e.g., Linux).

osversion
   Optional. Version of the operating system.

D. Additional sub-element to Target
-----------------------------------

A suggestion to add a "File" element was originally made in December
2000 by Andy Walther, that suggestion is being repeated here with some
modifications. The modifications to the original proposal are as follow:
the "hash" element is omitted; "name" is changed from an attribute to
a sub-element; a "category" attribute is added; an "fs_type" attribute
is added;  a "path" sub-element is added, an "accessed_time" sub-element
is added; the "size" element is split into two sub-elements: "data_size"
and "disk_size"; the "inode_number" sub-element is expanded and renamed
to "Inode"; a "FileAccess" sub-element is introduced to provide a more
generic representation of the information that was in "owner", "group",
and "permissions"; "links" is expanded and renamed to "Linkage"; and
an "AdditionalData" element is added. If the hash element is used, it
should be noted that TripWire supports the use of multiple types of
file hashes, potentially used in combination.

An attempt has been made here to define a "File" element that largely
transcends operating system (OS) implementations. However, a strict
adherence to OS independence would yield an data structure that lacks
information needed by downstream Managers, such as correlation systems.
The FilePermissions class in Java is an example of the type of data
structure that results from a strict adherence to total OS independence.
While such a data structure makes sense for Java, in IDMEF it creates
an opportunity for vendor-specific variety in Alert formats that is
not necessarily desirable. For that reason, there are some features
of the "File" element and its sub-elements that reflect the
representation
of information on a particular OS. One example of such OS dependence
is the "Inode" sub-element, which is particular to Unix, another
example is the "reparse point" attribute type in the "Linkage" element,
which is particular to NTFS. In all such cases, the associated XML
object
is optional and should be omitted when not relevant.

The proposal is to add zero or more "File" elements to "Target", as
defined below. This addition would standardize representation of file
information generated by host-based intrusion detection systems, as
noted above.

In XML the "File" element would be added to "Target" as follows:

<!ELEMENT Target (
    Node?, User?, Process?, File*, Service?
)>

++++ File element ++++

The File element could then be represented in XML as follows:

<!--
 | Values for the File.category attribute.
 -->
<!ENTITY % attvals.filecat              "
    ( original | current )
  ">

<!ELEMENT File                       (
    name, path, create_time?, modified_time?, accessed_time?,
    data_size?, disk_size?, FileAccess*, Linkage*, Inode?,
    AdditionalData?
  )>
<!ATTLIST File
    ident               CDATA                  '0'
    category            %attvals.filecat;      #REQUIRED
    fs_type             CDATA                  #REQUIRED
    %attlist.global;

The "File" element consists of several sub-classes:

name
   One. STRING. Name of the file to which the alert applies. This name
   does not include the path.

path
   One. STRING. Path, including the file name, to this file.
   Note that the use of separate name and path is used to be consistent
   with "Process" and to emphasize that path information is important.
If
   the path field is eliminated, the name information should include the
   path in order to avoid ambiguity. The path name used should be as
   universal a name as possible. Providing a more universal name will
   facilitate processing of the alert by alert correlation systems.

   For Windows, the path should be specified using the Universal Naming
   Convention (UNC) for files that are remote and using a drive letter
   for files that are local (e.g., C:\boot.ini). For Unix, if the file
   system is imported from another server (e.g., nfs), the name of the
   resource mounted should be used instead of the mount point. The name
   of the resource will include the name of the server and the directory
   on that server. The mount point can be provided using the "Linkage"
   element.

create_time
   Zero or one. DATETIME as described in Section 4.4.6 of
   draft-ietf-idwg-idmef-xml-04.txt. Create time of this file.

modified_time
   Zero or one. DATETIME as above. Modification time of this file.

accessed_time
   Zero or one. DATETIME as above. Time the file was last accessed.

data_size
   Zero or one. INTEGER. The size of the data, typically what is meant
   when referring to file size. On UNIX UFS, this value corresponds to
   st_size in struct stat. On Windows XP NTFS, this value corresponds
   to VDL.

disk_size
   Zero or one. INTEGER. The physical space taken up by the file on
disk.
   On UNIX UFS, this value corresponds to 512*st_blocks from struct
stat.
   On Windows XP NTFS, this value corresponds to EOF.

FileAccess
   Zero or more. Access settings on the file.

Linkage
   Zero or more. File system objects to which this file is linked.

Inode
   Zero or one. Inode information for this file, relevant for UNIX.

The "File" element has three attributes:

ident
    Optional. A unique identifier for the file.

category
    Required. The context for the information being reported.

    Rank  Keyword            Description
    ----  -------            -----------
     0    original           The file information is from prior to the
                             reported change.
     1    current            The file information is from after the
reported
                             change.

fs_type
   Required. Name of the file system type. The name should be specified
   as a standard abbreviation, for example: "ufs", "ntfs", "fat",
"vfat",
   "cdfs", "joliet", "udf", "pcfs", "tmpfs", etc. A string is used to
   improve extensibility. This attribute can be used by a Manager to
   govern how the elements should be interpreted. Note that if a file
   system is mounted from a remote host the type may be "nfs", "smbfs",
   etc.

++++ FileAccess element ++++

The "FileAccess" element represents the access policy of the file.
The representation of the access policy is intended to be valid
across operating systems. The "FileAccess" element is represented
in XML as follows:

<!ELEMENT FileAccess (
    ( UserId | GroupId ), permission*
 )>

The "FileAccess" element consists of three sub-classes:

UserId
   Zero or more. User to which the permissions apply.
   If the attributes are provided, they should be ignored.

GroupId
   Zero or more. Group to which the permissions apply.

permission
   Zero or more. STRING. Level of access allowed. Recommended values
   are: "noAccess", "read", "write", "execute", "delete",
   "changePermissions", "takeOwnership", and "executeAs". The
   "changePermissions" and "takeOwnership" strings represent those
   concepts in Windows. On Unix, the owner of the file always has
   "changePermissions" access, even if no other access is allowed
   for that user. 'Full Control' in Windows is represented by
   enumerating the permissions available under 'Full Control'. The
   "executeAs" string represents the setuid and/or setgid settings
   in Unix. A string is used to allow easy extensibility.

++++ GroupId element ++++

The "GroupId" element represents group id information: name and/or id.
The "GroupId" element is very similar to the "UserId" element and is
represented in XML as follows:

<!ELEMENT GroupId    (
    name | number | (name, number)
 )>

The "GroupId" class is composed of two aggregate classes. The aggregate
classes that make up the GroupId are:

name
    Zero or one. STRING. A user or group name. To represent the Unix
    permissions for "other", the name "unix_other" should be used.

number
    Zero or one. INTEGER. A user or group number.

The "GroupId" class has no attributes.

++++ Inode element ++++

The "Inode" element represents the i-node information found on the
UNIX file system. The i-node information tuple of number and device
id uniquely identifies a file on the file system.  Consequently,
it is useful for enabling a manager to determine when files are
hard links. Since the device id is derived from the major and minor
device numbers, only the latter are supplied in the "Inode" element.
The "Inode" element is represented in XML as follows:

<!--
<!ELEMENT Inode                       (
  )>
<!ATTLIST Inode
    number            CDATA      #REQUIRED
    major_device      CDATA      #REQUIRED
    minor_device      CDATA      #REQUIRED
    c_major_device    CDATA      #IMPLIED
    c_minor_device    CDATA      #IMPLIED

The "Inode" element has five attributes:

number
   Required. The inode number.

major_device
   Required. The major device number.

minor_device
   Required. The minor device number.

c_major_device
   Optional. The major device number if a character special file.

c_minor_device
   Optional. The minor device number if a character special file.

++++ Linkage element ++++

The "Linkage" element represents file system connections between
the file represented by the "File" element and other objects on
the file system. For example, if the "File" is a symbolic link
or shortcut then the "Linkage" element should contain the name
of the object that the link points to. Additional information
about the object identified in the "Linkage" element can be
supplied, if appropriate, using another "File" element.
The "Linkage" element is represented in XML as follows:

<!--
 | Values for the Linkage.category attribute.
 -->
<!ENTITY % attvals.linkcat              "
    ( hard link | symbolic link | mount point | reparse point |
      shortcut | stream )
  ">

<!ELEMENT Linkage                    (
    (name, path) | File
  )>
<!ATTLIST Linkage
    category            %attvals.linkcat;      #REQUIRED
    %attlist.global;

The "Linkage" element consists of three sub-classes:

name
    One. STRING. The name, without the path, of the link.

path
    One. STRING. The path, including the name, of the link.
    See the note on "File.path".

File
    One. Additional information about the object named in
    the Linkage. For example, if the "Linkage" "category" is
    "reparse point", the "File" information would include the
    name of the associated filterDriver; if the "Linkage"
    "category" is "stream", the "File" information would contain
    details on that stream.

The "Linkage" element has one attribute:

category
    Required. The type of link.

    Rank  Keyword            Description
    ----  -------            -----------
     0    hard link          A hard link. The "name" class represents
                             another name for this file. This
information
                             may be more easily obtainable on NTFS than
                             on other file systems.
     1    symbolic link      A symbolic link. The "name" class
represents
                             the file to which the File points. The
entity
                             represented by this File element is the
symbolic
                             link.
     2    mount point        Alias for the directory specified as part
of
                             the "path" and "name" combination.
     3    reparse point      Applies only to Windows, excludes symbolic
                             links and mount points, which are specific
                             types of reparse points.
     4    shortcut           File represented by the Windows "shortcut".
                             A shortcut is distinguished from a symbolic
                             link because of difference in their
contents
                             that may be important to a Manager.
     5    stream             Alternate Data Stream (ADS) in Windows,
fork
                             on MacOS. Separate file system entity that
is
                             considered an extension of the main "File".

Example Alert
=============

A. BIDS race condition alert using draft-ietf-idwg-idmef-04.txt.
----------------------------------------------------------------
Notes: IP addresses, NTP timestamp, and other numeric values are bogus.
       A | appears at the beginning of lines that will be affected by
       the proposed modifications.

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE IDMEF-Message PUBLIC "-//IETF//DTD RFCxxxx IDMEF v0.5//EN"
 "idmef-message.dtd">

 <IDMEF-message version="0.5">
|  <Alert impact="bad-unknown">
     <Analyzer analyzerid="bids-123.123.123.123">
|      <Node category="unknown">
         <name>mcs-001</name>
         <Address category="ipv4-addr">
           <address>123.123.123.123</address>
         </Address>
       </Node>
       <Process>
         <name>bids</name>
         <pid>12345</pid>
         <path>/usr/local/sbin</path>
         <arg>-r</arg>
       </Process>
     </Analyzer>
     <CreateTime ntpstamp="0x12345678.0x87654321">
       2001-08-14T21:21:32Z
     </CreateTime>
     <DetectTime ntpstamp="0x12345678.0x87654321">
       2001-08-14T21:21:21Z
     </DetectTime>
     <Source spoofed="no">
       <Node>
         <location>console</location>
         <Address category="ipv4-addr">
           <address>123.123.123.123</address>
         </Address>
       </Node>
     </Source >
     <Target decoy="no">
       <Node>
         <location>local</location>
         <Address category="ipv4-addr">
           <address>123.123.123.123</address>
         </Address>
       </Node>
       <User category="os-device">
         <UserId type="original-user">
           <number>456</number>
         </UserId>
         <UserId type="current-user">
           <name>fred</name>
           <number>456</number>
         </UserId>
         <UserId type="user-privs">
           <number>456</number>
         </UserId>
       </User>
|      <Process>
|        <name>ln</name>
|        <arg>-s</arg>
|        <arg>/tmp/ksh</arg>
|        <arg>/tmp/acr00238483</arg>
|      </Process>
     </Target >
     <Classification origin="vendor-specific">
       <name>DOM race - Dynamic object monitoring has detected a race
condition 
exploit</name>
       <url>file://</url>
     </Classification>
   </Alert>
 </IDMEF-message>

B. BIDS race condition alert using draft-ietf-idwg-idmef-04.txt
   with modifications.
----------------------------------------------------------------
Notes: IP addresses, NTP timestamp, and other numeric values are bogus.
       A | appears at the beginning of lines affected by the proposed
       modifications.

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE IDMEF-Message PUBLIC "-//IETF//DTD RFCxxxx IDMEF v0.5-mod//EN"
 "idmef-message.dtd">

 <IDMEF-message version="0.5">
|  <Alert impact="file-created">
     <Analyzer analyzerid="bids-123.123.123.123">
|      <Node category="hosts">
         <name>mcs-001</name>
         <Address category="ipv4-addr">
           <address>123.123.123.123</address>
         </Address>
       </Node>
       <Process>
         <name>bids</name>
         <pid>12345</pid>
         <path>/usr/local/sbin</path>
         <arg>-r</arg>
       </Process>
     </Analyzer>
     <CreateTime ntpstamp="0x12345678.0x87654321">
       2001-08-14T21:21:32Z
     </CreateTime>
     <DetectTime ntpstamp="0x12345678.0x87654321">
       2001-08-14T21:21:21Z
     </DetectTime>
     <Source spoofed="no">
       <Node>
         <location>console</location>
         <Address category="ipv4-addr">
           <address>123.123.123.123</address>
         </Address>
       </Node>
     </Source >
     <Target decoy="no">
       <Node>
         <location>local</location>
         <Address category="ipv4-addr">
           <address>123.123.123.123</address>
         </Address>
       </Node>
       <User category="os-device">
         <UserId type="original-user">
           <number>456</number>
         </UserId>
         <UserId type="current-user">
           <name>fred</name>
           <number>456</number>
         </UserId>
         <UserId type="user-privs">
           <number>456</number>
         </UserId>
       </User>
|      <File category="current">
|        <name>acr00238483</name>
|        <path>/tmp/acr00238483</path>
|        <FileAccess>
|          <UserId><name>alice</name><number>777</number></UserId>
|          <permission>read</permission>
|          <permission>write</permission>
|          <permission>delete</permission>
|          <permission>changePermissions</permission>
|        </FileAccess>
|        <FileAccess>
|          <GroupId><name>user</name><number>42</number></GroupId>
|          <permission>read</permission>
|          <permission>write</permission>
|          <permission>delete</permission>
|        </FileAccess>
|        <FileAccess>
|          <GroupId><name>unix_other</name></GroupId>
|          <permission>noAccess</permission>
|        </FileAccess>
|        <Linkage category="symbolic">
|          <name>ksh</name>
|          <path>/tmp/ksh</name>
|        </Linkage>
|      </File>
     </Target >
     <Classification origin="vendor-specific">
         <name>DOM race - Dynamic object monitoring has detected a race
conditio
n exploit</name>
         <url>file://</url>
     </Classification>
   </Alert>
 </IDMEF-message>

---

From: "Laura S. Tinnel" <[email-masked]>
Date: Wed, 24 Oct 2001 11:46:48 -0400

Cynthia,

We have been using IDMEF to represent Tripwire and other host-based alerts, 
and I have several comments about your recommendations.

>1. Host-Based Intrusion Detection Systems

I've actually been quite pleased at the flexibility afforded by the 
AdditionalData field, and I do not necessarily see this as a problem as 
long as a data dictionary is used. If it is possible to extrapolate common, 
generic host-based sensor features, then it is useful to add such to the 
"IDMEF language". For information which is specific to a particular 
host-based sensor, the AdditionalData field combined with a data dictionary 
is the appropriate way to express such. I wanted to caution here because 
based on past experience, I'm concerned that the IDMEF specification will 
grow exponentially if the authors attempt to satisfy every possible 
specific use. See my response to the "impact" attribute comments.

>probably warrants more scrutiny but it is not addressed here beyond this

For this particular case, we simply ignored the "Source" field. This is 
because I got the sense that the Source/Target idea was designed with 
network-based sensors in mind and that Source just didn't apply. I think it 
would be helpful to have a better way to express the Source of a host-based 
attack, assuming that
the host sensor could deduce the Source, which Tripwire cannot. Audit trail 
analyzers could certainly tie the source of certain actions to a process 
and possibly a user account. It would probably prove useful to take a 
survey of the possible source values that host-based detectors could 
generate and then compare with what one can currently express using the 
Source field.

>A. Alert impact types:

I disagree with this use of the "impact" attribute. Impact should have some 
stronger meaning (although I'm not too enamored with the current use of 
"bad-unknown" and "unknown" either.) The impact is the results or 
consequences of the activity detected and is all a matter of perspective. 
 From a Tripwire perspective, all the keywords you have specified here are 
really actions or activities, not impacts. Tripwire detects the activity 
that a file has been created. What is the impact of the file creation? 
Unfortunately, Tripwire cannot tell you nor can it make a judgement of the 
goodness or badness of the file system change, thus the "impact" is 
"unknown". The problem is that "impact" really is contextual and depends on 
the level of the analyzer. Let's look at an example of some different 
levels of perspective:

Activity 1: httpd buffer overflow vulnerability exploited
Results 1: file created

Activity 2: file created
Results 2: web site defaced

Activity 3: web site defaced
Results 3: cannot conduct online trading

This is why having a free form string with the possible values defined by 
the analyzer creating the report is the right solution. I don't disagree 
that this kind of activity information should be expressed. It might be 
useful to add an <Activity> tag to the IDMEF specification for expressing 
info on the detected activity. For now, I simply used the AdditionalData 
field to express these things. I would also point out that there are other 
missing actions from your list. The Windows version of Tripwire also looks 
at registry changes, so one could also specify "registry-key-modified" and 
other such actions. The problem one gets into when trying to specify all 
the possible values is that each time the vendor updates their tool to have 
some new detection capability, the specification must be modified. The 
repercussions of such changes are much less drastic if the vendor only 
needs to update and distribute their data dictionary. Does this limit the 
native expressiveness of the IDMEF language? Yes. But such tradeoffs must 
be made.

As a side note, I did notice that an "action" attribute was added to allow 
an analyzer to specify actions it took in response to the detected 
activity. I've been doing some work on response activity reporting and am 
fairly confident that a single attribute will not suffice. I'm also not 
convinced that IDMEF is the right language for such reporting.

I do not have specific comments on the rest of your suggestions other than 
in a cursory look, they appear to be fairly generic and applicable to 
multiple host-based sensors. I'll try to look more deeply into them shortly.

-Laura