Subject: DTD/Data Model changes (Message 1 of 4)

From: "David A. Curry" <[email-masked]>
Date: Wed, 03 Jan 2001 13:59:00 -0500

The following is a list of changes that are being made to the IDMEF Data Model
and/or the IDMEF XML DTD, or to the Internet-Draft that describes them,
following the San Diego IETF/IDWG meetings.

 1. The merged Data Model/XML DTD Internet-Draft is being rearranged and parts
    of it rewritten so that it reads in a more logical way.  The tentative
    Table of Contents for the document is now:

    Status of This Memo
    1. Abstract
    2. Conventions Used in This Document
    3. Introduction
       3.1 The IDMEF Data Model
          3.1.1 Problems Addressed by the Data Model
          3.1.2 Data Model Design Goals
             3.1.2.1 Representing Events
             3.1.2.2 Content-Driven
             3.1.2.3 Relationship Between Alerts
       3.2 Implementing the IDMEF in XML
          3.2.1 The Extensible Markup Language
          3.2.2 Rationale for Implementing IDMEF in XML
    4. Notational Conventions and Formatting Requirements
       4.1 Universal Modeling Language
          4.1.1 Relationships
             4.1.1.1 Inheritance Relationship
             4.1.1.2 Aggregation Relationship
          4.1.2 Multiplicity Indicator
    4.2 Extensible Markup Language
          4.2.1 The IDMEF Document Prolog
             4.2.1.1 XML Declaration
             4.2.1.2 IDMEF DTD Formal Public Identifier
             4.2.1.3 IDMEF DTD Document Type Definition
          4.2.2 Character Data Processing in XML and IDMEF
             4.2.2.1 Character Entity References
             4.2.2.2 Character Code References
             4.2.2.3 White Space Processing
          4.2.3 Languages in XML and IDMEF
       4.3 IDMEF Data Types
    5. The IDMEF Data Model and DTD
       5.1 Data Model Overview
       5.2 The Core of the Data Model
          5.2.1 The IDMEF-MESSAGE Class
          5.2.2 The ALERT Class
          5.2.3 The ANALYZER Class
          5.2.4 The CREATETIME Class
          5.2.5 The DETECTTIME Class
          5.2.6 The ANALYZERTIME Class
          5.2.7 The CLASSIFICATION Class
          5.2.8 The SOURCE Class
          5.2.9 The TARGET Class
          5.2.10 The ADDITIONALDATA Class
          5.2.11 The TOOLALERT Class
          5.2.12 The CORRELATIONALERT Class
          5.2.13 The OVERFLOWALERT Class
          5.2.14 The Support Classes
             5.2.14.1 The IDENT Class
             5.2.14.2 The ADDRESS Class
             5.2.14.3 The USER Class
             5.2.14.4 The NODE Class
             5.2.14.5 The PROCESS Class
             5.2.14.6 The SERVICE Class
             5.2.14.7 The WEBSERVICE Class
             5.2.14.8 The SNMPSERVICE Class
    6. Extending the IDMEF
       6.1 Extending the Data Model
          6.1.1 Extension by Aggregation
          6.1.2 Extension by Subclassing
       6.2 Extending the DTD
    7. Special Considerations
       7.1 XML Validity and Well-Formedness
       7.2 Analyzer-Manager Time Synchronization
       7.3 NTP Timestamp Wrap-Around
       7.4 Unrecognized XML Tags
       7.5 Digital Signatures
    8. Examples
    9. The IDMEF XML Document Type Definition
    10. Security Considerations
    11. References
    12. Acknowledgements
    13. Author's Addresses
    Full Copyright Statement

 2. The existing <User> element and its sub-elements are being replaced with
    the new model defined by Glenn Mansfield and me, with modifications as
    suggested by Herve Debar, as proposed and agreed to on the idwg-public
    list in Oct/Nov 2000.

    This results in the following general format:

    <User category="unknown|application|os-device">
      <Id type="original-user|current-user|target-user|user-privs|
                    current-group|group-privs">
        <name>user name</name>
        <number>user id</number>
      </Id>
      ...
    </User>

 3. The representation of time is being simplified as proposed by Paul Sangree
    and agreed to in San Diego:

    a. Eliminate the <Time> element, and created a new <CreateTime> element
    b. Eliminate the <time> and <date> elements
    c. Date and time format to ISO 8601:2000 standard
    d. Change <ntpstamp> from an element to an attribute

    This results in the following general format:

    <Alert>
      <AnalyzerTime ntpstamp="0xBDFA4701.0x32C6">
        2000-12-25T01:00:01.15+0000
      </AnalyzerTime>
      <DetectTime ntpstamp="0xBDFA4701.0x32C6">
        2000-12-25T01:00:01.15+0000
      </DetectTime>
      <CreateTime ntpstamp="0xBDFA4701.0x32C6">
        2000-12-25T01:00:01.15+0000
      </CreateTime>
    </Alert>
   
    Language will be added to the Internet-Draft to cover formatting the
    date/time strings according to ISO 8601:2000, and to cover some subleties
    of using the NTP timestamp.

 4. Support for isolated networks and sensors with multiple interfaces is
    being added, as proposed by Paul Sangree and agreed to in San Diego:

    a. Add an optional <interface> attribute to <Source> and <Target> which
       can be used to identify the interface on which a network sensor saw the
       traffic.
    b. Add optional attributes "vlan-num" and "vlan-name" to the <Address>
       element.

 5. The <Environment> and <Argument> elements are being removed, moving <env>
    and <arg> up one level.  Proposed by Paul Sangree and agreed to in San
    Diego.

 6. The value "unknown" is being removed from the list of possible values for
    the "type" attribute on <AdditionalData>, as it makes no sense.  Proposed
    by Paul Sangree and agreed to in San Diego.

 7. The content model for <AdditionalData> is being changed from "#PCDATA" to
    "ANY", and added "xml" as a possible value for the "type" attribute.

    The purpose of this is to fix the extensibility problems discussed in San
    Diego.  By making this change, we can allow people to include additional
    DTDs in their IDMEF markup (e.g., one for packet headers), and to put all
    the new markup underneath <AdditionalData type="xml">.  This change will
    also allow us to make better (i.e., correct) use of XML Namespaces.

    The text in previous Internet-Drafts tells people how to add or change
    elements in their IDMEF Messages.  All of that language is being removed,
    and replaced with new requirements that basically say that the only way
    you can do extensions is by including new DTDs, and anything you include
    gets put under <AdditionalData type="xml">.  Futhermore, any extensions
    you do add must use a different XML Namespace (i.e., they can't use
    "idmef" or the default namespace) to avoid conflicts with existing IDMEF
    elements and attributes.

    I believe this change also addresses the issues raised on the list by Tara
    Whalen (including data that uses a radically different data model, such as
    anomaly data) and Joe McAlerney (include packet header data).  To do this,
    just write a DTD for the data you want to include, and put the data (with
    all your new tags) under <AdditionalData>.  You get the data in the alert,
    and any managers that don't know what to do with it can just throw it
    away; the rest of the IDMEF format does not change.

    *** NOTE: I was tasked with investigating this problem in San Diego; the
	above is my solution.  We are limited in how we can solve the problem,
	mostly because of the limitations imposed by DTDs.  XML Schemas will,
	I think, allow us a more "elegant" solution, but they are currently
	only in Candidate Recommendation status within the W3C, which in
	effect means we'll be unable to use them until "Version 2" of IDMEF,
	given our current timetable.

 8. The caveats on <AnalyzerTime> and time syncrhonization made in Paul
    Sangree's presentation in the appropriate section(s) of the
    Internet-Draft, are being included, as agreed to in San Diego.

    Also, some recommendations on how implementers should handle the
    time synchronization problem are being added.

 9. The <portlist> element is being removed from the <Service> element.  To
    achieve lists of ports, specify multiple <Target> elements.

--- end of message ---

---

From: Stuart Staniford <[email-masked]>
Date: Mon, 15 Jan 2001 17:00:39 -0800

Everything looks great except:

>     achieve lists of ports, specify multiple <Target> elements.

I've given this a little thought, and I'm not sure this is a good change. 
The old definition was  

> 7. Port lists, as used in the <portlist> element, are encoded as a 
>       comma-separated list of numbers (individual integers) and ranges 
>       (N-M means ports N through M, inclusive). Any combination of 
>       numbers and ranges may be used in a single list. For example: 5-
>       25,37,42,43,53,69-119,123-514. 

and an example in the document was

>           <Target> 
>             <Node category="dns"> 
>               <name>www.bigcompany.com</name> 
>               <Address category="ipv4-addr"> 
>                 <address>123.234.231.121</address> 
>               </Address> 
>             </Node> 
>             <Service> 
>               <portlist>5-25,37,42,43,53,69-119,123-514</portlist> 
>             </Service> 
> 	  </Target>

If we get rid of portlist, this will become

           <Target> 
             <Node category="dns"> 
               <name>www.bigcompany.com</name> 
               <Address category="ipv4-addr"> 
                 <address>123.234.231.121</address> 
               </Address> 
             </Node> 
             <Service> 
               <dport>5</dport> 
             </Service> 
 	  </Target>
          <Target> 
             <Node category="dns"> 
               <name>www.bigcompany.com</name> 
               <Address category="ipv4-addr"> 
                 <address>123.234.231.121</address> 
               </Address> 
             </Node> 
             <Service> 
               <dport>6</dport> 
             </Service> 
 	</Target>
          <Target> 
             <Node category="dns"> 
               <name>www.bigcompany.com</name> 
               <Address category="ipv4-addr"> 
                 <address>123.234.231.121</address> 
               </Address> 
             </Node> 
             <Service> 
               <dport>7</dport> 
             </Service> 
 	</Target>

	... continue for another five hundred or so ports ...

Did I misunderstand?

Are you sure that's a good idea?

Stuart.

-- 
Stuart Staniford  ---  President  ---  Silicon Defense
[email-masked]  http://www.silicondefense.com/
(707) 445-4355                     (707) 445-4222 (FAX)

---

From: "David A. Curry" <[email-masked]>
Date: Tue, 16 Jan 2001 08:03:51 -0500

>If we get rid of portlist, this will become

[...]

>Are you sure that's a good idea?

Your point is well taken, and I had the same thought -- I'm not sure it's a
good idea.  I had hoped to see some discussion of it on the list.

I had two main thoughts when I decided to remove it:

1. The "trend" at the last meeting, on several folks' part, was to "flatten
   out" the XML -- i.e., it's preferable to have multiple instances of some
   basic elements to provide multiple pieces of data, rather than building
   all these big "containers" for multiple data items.

2. Because of where <Service> lives (inside <Target> but not inside <Source>),
   you can use <portlist> to represent a list of target ports, but not a list
   of source ports.  Really, I think you should be able to use it for both.

I'm willing to be convinced that I'm wrong and put <portlist> back in.  But I
think what I'd really like to see is a combination of <portlist> and Paul
Sangree's proposed change to remove <sport> and <dport> from <Service> and put
a <port> element in <Source> and <Target>.

Instead of <port>, I would propose that we use <ports count="n"> instead.  If
"count" is 1 (the default), a single port is listed.  If "count" is more than
1, then the old <portlist> syntax applies.  And because it would now be in
<Source> and <Target>, you can specify multiple ports on either end.

Comments?

--Dave