Subject: [IPFIX] Fwd: New Version Notification for draft-trammell-ipfix-ie-doctors-00

From: Brian Trammell <[email-masked]>
Date: Fri, 1 Oct 2010 14:15:21 +0200

Greetings, all:

A first draft of the IE-Doctors draft discussed in Maastricht has been posted; this presents a reasonably complete discussion of the issues surrounding efficient extension of the Information Element registry and definition of new IPFIX applications in Internet-Drafts. We intend to discuss this as a future BCP at the meeting in Beijing. Reviews and comments in the meantime are of course appreciated. :)

Best regards,

Brian

Begin forwarded message:

> From: IETF I-D Submission Tool <[email-masked]>
> Date: October 1, 2010 2:01:48 PM GMT+02:00
> To: [email-masked]
> Cc: [email-masked]
> Subject: New Version Notification for draft-trammell-ipfix-ie-doctors-00 
> 
> 
> A new version of I-D, draft-trammell-ipfix-ie-doctors-00.txt has been successfully submitted by Brian Trammell and posted to the IETF repository.
> 
> Filename:	 draft-trammell-ipfix-ie-doctors
> Revision:	 00
> Title:		 Guidelines for Authors and Reviewers of IPFIX Information Elements
> Creation_date:	 2010-10-01
> WG ID:		 Independent Submission
> Number_of_pages: 22
> 
> Abstract:
> This document provides guidelines for the definition of IPFIX
> Information Elements for addition to the IANA IPFIX Information
> Element registry, in order to extend the applicability of the IPFIX
> protocol to new operations and management areas.
> 
> 
> 
> The IETF Secretariat.
>

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 7 Oct 2010 19:05:33 -0400

Hi Brian,
I read this draft, and most of it seems good/useful/motherhood-and-apple-pie.

The only substantive comment/question I have is on section 9, the textual IESpec stuff.  Do we really need yet another textual representation of meta-data?  Isn't there already an XML-based one from rfc5102?  

I note your statement that the design goal was compactness and not needing XML support, but it's not clear why those properties matter when the meta-data is never on the wire in its textual form.  Don't get me wrong, I'm not an XML fan by any stretch, but I'd hate to have two textual representation formats for one protocol, and re-inventing the wheel with yet-another-new-format seems silly. (Nevermind that the IETF already uses SMI/ASN.1 for SNMP, XML for DIAMETER, ABNF for a HTTP/SIP/RTSP/SMTP/etc., defacto-standards for RADIUS and DNS, and who knows what else. :)

It's also kinda weird to have a new text format in a Guidelines doc.

-hadriel

On Oct 1, 2010, at 8:15 AM, Brian Trammell wrote:

> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Fri, 8 Oct 2010 08:39:41 +0200

Hi, Hadriel,

Thanks for the feedback.

The primary design goal of the IESpec format is human readability in the expression of information model extensions and templates, while maintaining simple machine readability. The idea is implementors can read an Internet Draft about an IPFIX application with example templates in IESpec form, copy and paste the example templates out of it into an implementation supporting IEspec, fill in the fields and go. For readability and compactness, it's intended to address shortcomings both in the XML format(s) already defined for IPFIX IEs, and in the traditional 32-bit-wide box-and-line diagrams, which make it difficult to fit a nontrivial template onto a single page, and require some manual work to turn into an implementation.

The XML format in 5102 has been partially superceded by the one used by IANA internally (I'm not sure the extent to which the one uses the elements from the other, it's been a while), and is specifically targeted only for defining Information Elements to be added to the registry. (If you need to put IE metadata on the wire, e.g. to define enterprise-specific IEs in long-term storage, 5610 covers that.)

As to the structure, you have a point; it's a little tacked on. IESpec was formerly (well, currently; it's not expired yet) its own document, and integrated into ie-doctors on several suggestions in Maastricht; I'm not yet decided where it belongs...

Cheers,

Brian

---

From: Hadriel Kaplan <[email-masked]>
Date: Fri, 8 Oct 2010 15:27:03 -0400

Hi Brian,
comments inline...

On Oct 8, 2010, at 2:39 AM, Brian Trammell wrote:

> Hi, Hadriel,
> 
> Thanks for the feedback.
> 
> The primary design goal of the IESpec format is human readability in the expression of information model extensions and templates, while maintaining simple machine readability. The idea is implementors can read an Internet Draft about an IPFIX application with example templates in IESpec form, copy and paste the example templates out of it into an implementation supporting IEspec, fill in the fields and go.

At the end of the day what we're talking about is a form of a "dictionary" file, similar to those used for other protocols, like for SNMP, DIAMETER, RADIUS, etc.  Right?  Like not just for putting into a RFC, but more importantly for importing into a reader so it can decode and understand the IEs, right?

Again, I'm not a big fan of XML, but there are a ton of XML parser libraries available, and the importing of that into a reader so it can understand the new IEs is done rarely, and with a CPU (not an FPGA or network processor).  Libraries exist to not only read them, but verify their correctness using a schema.  Further, they support comments and text descriptions, which are useful for an IPFIX reader importing them.  An IPFIX reader can use those to display in a GUI as additional help, and a human can use them to understand stuff when he/she reads the XML.  

For example, suppose vendor BigCorp wants to give out such a "dictionary" file for some PEN IEs its BigBox generates, so that IPFIX Readers can decode them.  One of those PEN IEs is for a "camFlowId".  In IESpec, it might be:
camFlowId(35566/123)<unsigned32>[4]

That doesn't tell the human nor the IPFIX collector/reader much info.  In XML it's a lot longer text, but it can include some more info like this:
       <description>
         <paragraph>
           An identifier of a single Content Addressable Memory index allocated for the flow.  Two CAM entries are allocated, for each direction, unless the system is provisioned otherwise.
         </paragraph>
       </description>
       <reference>
         <paragraph>
         See BigCorp User Manual chapter titled "Hardware Resources" for more information.
         </paragraph>
       </reference>

And the file can even have a "<-- BigCorp dictionary file for BigCorp BigBox 10000 System, version 1.2.3. Created 2-10-2010 -->" comment at the top.

> For readability and compactness, it's intended to address shortcomings both in the XML format(s) already defined for IPFIX IEs

What shortcomings?  There're some fields I think it should have that it doesn't, and the structure of it is weird, but  I'm a nitpicker.  Is there something fatally wrong with it?

> and in the traditional 32-bit-wide box-and-line diagrams, which make it difficult to fit a nontrivial template onto a single page, and require some manual work to turn into an implementation.

This is one of the things I was worried about - that we would be saying this would avoid showing things in 32-bit-wide box-and-line diagrams.  I find those drawings are really, really useful.  For example, I couldn't make heads or tails of the structured-data draft until I saw those examples.  

Fitting them on a page is not a serious issue really, is it?  I mean if it crosses pages, people can deal.  I know it takes more work to draw them, but you're writing an RFC for others to be able to interop with you, so that's life.  If a nontrivial template is difficult to draw, maybe that will encourage people to not make complicated templates. :)  I'm half joking, but my point is the work to write the RFC is really moot compared to helping people reading the RFC implementing it right.  

> The XML format in 5102 has been partially superceded by the one used by IANA internally (I'm not sure the extent to which the one uses the elements from the other, it's been a while), and is specif…

Right, I know 5610 puts the meta-data on the wire, and IESpec looks to me like basically just a text transformation of that.  But I see 5610 as a limited functionality, last-resort solution, in case the reader/collector has no real dictionary file available. (it seems odd to me that there's really a use-case for that in practice, but that's just me)  The point is because it's on-the-wire it has to be terse and limited, but a real dictionary file shouldn't be.

-hadriel

---

From: Brian Trammell <[email-masked]>
Date: Mon, 18 Oct 2010 09:21:24 +0200

Hi, Hadriel,

Comments per tradition inline...

> At the end of the day what we're talking about is a form of a "dictionary" file, similar to those used for other protocols, like for SNMP, DIAMETER, RADIUS, etc.  Right?  Like not just for putting…

There are two orthogonal dimensions here, on both of which we're getting confused... First there's the human-readable versus machine-readable issue, then the dictionary (Information Model) version template issue. On reflection, the primary application for iespec (and therefore the reason to include it in iedoctors) is human-readable templates with easy machine readability.

As a dictionary file, it's basically a human-editable subset of 5610, and as such only useful for the <type> and name information. But as a dictionary, the IANA registry is of course far preferable. So if the collector device is capable of doing so, I'd recommend downloading the IANA registry and parsing it at boot or pre-parsing it (maybe into 5610)  once per fortnight or so. As long as I don't have to _hand edit_ the IANA XML, I'm happy.

> What shortcomings?  There're some fields I think it should have that it doesn't, and the structure of it is weird, but  I'm a nitpicker.  Is there something fatally wrong with it?

Eyeball-parsing the XML formats for dictionary purposes is daunting, and they're not at all adaptable to easily readable templates. Fortunately, IANA has the eyeball parsing problem fixed, 

> This is one of the things I was worried about - that we would be saying this would avoid showing things in 32-bit-wide box-and-line diagrams.  I find those drawings are really, really useful.  For…

The thing is, they're useful primarily when describing the _framing_ of the protocol. Here's what a template looks like. Here's what a message looks like. Given a template, here's a record. Structured data is defining new _types_, with internal structure, and displaying that structure. Of course we need 32-bit box and line diagrams in that case. With respect to simple applications of IPFIX, just defining a new template doesn't change the framing. But I hadn't considered the possibility that I'm old and jaded, and that everyone else enjoys seeing the same boxes I'm getting bored with. :) So that's a valid argument against more compact templates in drafts...

The structured-data heirarchy extensions for IEspecs are meant to illustrate applications of structured-data, not the framing thereof. But there's a very valid point here -- structured-data is new enough that we really probably _do_ want to draw things out explicitly the first few times we use it.

> Fitting them on a page is not a serious issue really, is it?  I mean if it crosses pages, people can deal.  

A thirty-element template (not unthinkable) crosses two pages. At some point there's a tradeoff between verbosity for clarity and so much verbosity that you need to tape things to the wall to have any hope of seeing what's going on. 

> I know it takes more work to draw them, but you're writing an RFC for others to be able to interop with you, so that's life.  If a nontrivial template is difficult to draw, maybe that will encourage people to not make complicated templates. :)  I'm half joking, but my point is the work to write the RFC is really moot compared to helping people reading the RFC implementing it right.  

I chose the word "non-trivial" poorly -- there's not really a complexity continuum in templates, it's either "long" or "short". It's hard to draw _long_ templates, and it's a bad idea to encourage people to use short templates when what they need is a long one. :)

The reason to reduce the amount of work to write an RFC, though, is not to make _my_ life easier. It's to make it easier for IPFIX _non_-experts to write RFCs describing new applications. Less work, and less chance of making a mistake leading to ambiguity, means better, more readable, more quickly and easily edited and published RFCs.

Regards,

Brian

---

From: Hadriel Kaplan <[email-masked]>
Date: Tue, 19 Oct 2010 10:05:37 -0400

Hi Brian,
I agree with your comments.  I find the IESpec short-hand especially useful for emails more than anything else, so even for just that I'm fine with 'em. :)
I do worry about having two forms of dictionaries for IPFIX readers, but I suppose the market will sort it out. (or not)

-hadriel