Subject: [OPSAWG] comments/edits on pcapng

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Nov 2015 20:05:25 -0500

11hr trips home are good for document review...
Is the XML around? Found it: https://github.com/pcapng/pcapng/pull/38

I reviewed draft-tuexen-opsawg-pcapng, while it hasn't had an -01, some work
has been occuring in the github copy, and so the effort isn't dead, just not
as visible.

For those that don't know, I'm been the packager/maintainer for tcpdump and
libpcap for about the last 15 years, and until 5-6 years ago, pretty much
anyone that used *pcap* format, used libpcap to read/write it.
That has changed (which is good!) since the advent of the pcapng format.
I personally didn't have a lot of time to review the ideas when they were in
their formative stage.  Aside from my initial whine, I think the document is
quite mature, and I'd like to see it move forward, either within opsawg, or
as ISE.

Tcpdump has been able to read pcapng files for awhile now, but still writes
pcap format files.

This is definitely not my first read of the document, but the idea of going
via OPSAWG surfaced at IETF90 (Toronto), but seems to have gotten stuck?

In general, I had thought that pcapng was going to start with an identical
magic as pcap, but that the "version_major" would say, "2", i.e:
pcap1:	bpf_u_int32 magic = 0xa1b2c3d4  (mutated for byte order)
	u_short version_major;
	u_short version_minor;

when in fact, pcapng files start with:
     int32_t type = 0x0A0D0D0A
     int32_t length;
     int32_t byte_order_magic;
     int16_t major; int16_t minor;

and the initial block isn't any different than any other blocks.  I see the
logic for this, and I don't have a problem with this on account of working,
deployed code, I just wish they could all have been "pcap" files.  So this is
more a whine than a comment.

I also find "pcapng" a poor long-term name, mostly it's the "ng" part that I
dislike. .pc2p or something might have been better.

section 4.1 says this about major/minor:

        Major Version: number of the current mayor version of the
              format. Current value is 1. This value should change if the format changes in
              such a way that code that reads the new format could not read the old format
              (i.e., code to read both formats would have to check the version number and
              use different code paths for the two formats) and code that reads the old
              format could not read the new format.

        Minor Version: number of the current minor version of the
              format. Current value is 0. This value should change if the format
              changes in such a way that code that reads the new format could read
              the old format without checking the version number but code that
              reads the old format could not read all files in the new format.

and I think that I don't buy it.
[First, "major version" should have been started at "2" :-) ]

But, more importantly, we have an easily, and almost always incremenally
extendable file format.  It's hard to imagine a change so drastic that it
would be worth incrementing the major version number that would still leave
enough of the file format intact that there would be any value left.
One would also wind up changing magic/header numbers.
With protocols that are online, it is different, as there can be a bit of
negotiation when major numbers don't match, but not really the case for files
resting on disk.

So I think that we should simply turn this into a 4-byte int32 field.  Let's
just put the highest *RFC* number that must be understood in order to comprehend
all the blocks present.  At approx. 3500 RFCs/10 years, 32-bits will last a
long long long time... We could even do this with just the Minor Version and
be good until rfc65535, which might be by oh year 2265 assuming the IETF
continues at it's present rate.
Maybe we could have a bit at the top which says, "There are experimental
blocks, and understanding them is critical to the semantic content of file"

On the Interface Description Block, (section: 4.2), the table shown should be
an IANA controlled registry.  The reference to section 3.5 is initially
confusing when reading.  I think that there should be a single table (in
section 3.5).  I see that the Section Header Block also reuses codes used in
the IDB:  I think that this is a mistake, the codes in a comment block should
not be context dependant.   My pull request takes care of all of them, I
hope.  I numbered them sequentially, but we could leave gaps.

We will need a registration formula for this registry, and I think that
First-Come / First Served (RFC5226/BCP26, section 4.1) will do fine for this one.

On the subject of the Encryption Block.  In discussion on the list many asked
if we needed it.  Let me suggest that we do, but that yes, we need more
information in it.  First, we need to make it look more like ESP (RFC4303):

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ----
|               Security Parameters Index (SPI)                 | ^Int.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Cov-
|                      Sequence Number                          | |ered
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ----
|                    Payload Data* (variable)                   | |   ^
~                                                               ~ |   |
|                                                               | |Conf.
+               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Cov-
|               |     Padding (0-255 bytes)                     | |ered*
+-+-+-+-+-+-+-+-+               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |   |
|                               |  Pad Length   | Next Header   | v   v
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ------
|         Integrity Check Value-ICV   (variable)                |
~                                                               ~
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

But, we don't need a SeqNumber, or a Next Header.  We can probably assume
that the things inside are already padded to 32-bit boundary, but
self-describing padding is easy and simple.  Yes, we should have an ICV.
I make the SPI 64-bits to give it more space.  Who can decrypt and
even with what algorithm is actually unimportant/private.  If you have
to ask, then it's not you!  Further a comment in the SHR would permit
proper document management, if the file was part of an evidence locker.

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Security Parameters Index (SPI -- 64 bits)            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Payload Data* (variable)                   |
~                                                               ~
|                                                               |
+               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|               |     Padding (0-255 bytes)                     |
+-+-+-+-+-+-+-+-+                             +-+-+-+-+-+-+-+-+-+
|                                             |    Pad Length   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Integrity Check Value-ICV   (variable)                |
~                                                               ~
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Now, about LinkType: I suggest that having libpcap/tcpdump continue to
maintain the table is not ideal.  Instead, let's create an IANA registry for
it, populate it with existing values, and make the libpcap group the Expert
Review.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Guy Harris <[email-masked]>
Date: Tue, 10 Nov 2015 19:24:42 -0800

> has been occuring in the github copy, and so the effort isn't dead, just not

The official version of the spec is, indeed draft-tuexen-opsawg-pcapng.xml at the GitHub repository.

> Tcpdump has been able to read pcapng files for awhile now, but still writes

...as is the case with other programs using libpcap.

The current capabilities are what can be done without any additions to the API.  At some point I plan to add additional APIs to expose the full capabilities of pcapng when reading captures and to write pcapng files.  The APIs for reading files will also support pcap files, so if a program such as tcpdump is changed to use the new APIs when available, it'll still handle pcap files.

> enough of the file format intact that there would be any value left.

Well, one change that would be that drastic would be changing it from a format in which each block type has its own name space for options to a format in which there's a single global name space for options, and thus changing the option numbers.  That change would not only mean that old code would no longer be able to read the new files, it would also mean that code that handles the new files would need a different code path to handle the old files.

It would not, however, change the format of the blocks, for example, so the rest of the code, with the exception of the option processing code, would need no change (and the option processing code would only need to handle the renumbering of the options).

> One would also wind up changing magic/header numbers.

The main magic number is the byte-order magic number, which would only need to change if the format of the SHB changed.

> continues at it's present rate.

So are you suggesting that updates to the format, such as the addition of new blocks, new options, new link-layer types, etc. would be done with RFCs?

That would fit with the use of IANA registries for block types/option values/link-layer header types/etc..

> an IANA controlled registry.

I.e., replace http://www.tcpdump.org/linktypes.html with an IANA registry?

> The reference to section 3.5 is initially
> hope.

Does your pull request also bump the current major version number?  Renumbering the options means that all the code out there that reads the current file formats won't be able to read the new format *and* code that *does* read the new format would need to interpret the option values differently based on the version number, so that sure sounds like a major version change.

I don't see what the problem is with different block types having different option name spaces.  Common code to handle options in all block types wouldn't be that useful - that code wouldn't, for example, be able to do anything useful with the interface speed option in a Name Resolution Block.  (And what do you mean by a "comment block"; we don't have blocks whose sole purpose is to contain comments, we have a comment option that can be used in any block, the content of which is solely intended for display to humans.)

> information in it.

Well, the first question is what sort of information would be put in it.  The current spec just says it "is used to store encrypted data" and says of the data in it that "Once decrypted, it originates other blocks.", but it doesn't say what sort of stuff is being encrypted!  Lists of credit card numbers? :-)

> it, populate it with existing values, and make the libpcap group the Expert

So where would the actual formats be made available?

If we look at, for example, the snoop data link types registry:

	http://www.iana.org/assignments/snoop-datalink-types/snoop-datalink-types.xhtml#snoop-datalink-types-2

it just gives the code, a name for the type, and a reference, which links to RFCs as a reference.

If we were to publish an RFC for each linktype, or possibly incorporate a number of linktypes in a single RFC so as not to waste an entire RFC to say "IEEE 802.3 Ethernet" for type 1, that would work - in effect

	http://www.iana.org/assignments/pcap-datalink-types/pcap-datalink-types.xhtml

would replace the "LINKTYPE_ value" and a somewhat more human-oriented version of the "LINKTYPE_ name" column, and point to RFCs that provide information now in the "Description" column, of http://www.tcpdump.org/linktypes.html.

I don't think an RFC is required as a reference, so we wouldn't have to wait for an RFC to percolate through the process before making a new link-layer header type available, but that would leave open the question of what to use as a reference until the RFC comes out, and if you don't want www.tcpdump.org to store the description until it shows up in an RFC...

...well, actually I guess we could use the pcap-ng GitHub repository for that.

---

From: Warren Kumari <[email-masked]>
Date: Thu, 12 Nov 2015 00:42:39 +0900

> has been occuring in the github copy, and so the effort isn't dead, just not

I'm glad to hear this - it seemed like something worth pursuing - I
had a todo that got buried under other things to followup with the
authors to see whatever happened.

> quite mature, and I'd like to see it move forward, either within opsawg, or

The IETF is traditionally not very good at formats and such, but I'm
game to try progress it under OpsAWG.
W

> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg
>

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Michael Richardson <[email-masked]>
Date: Thu, 12 Nov 2015 11:26:36 -0500

Guy Harris <[email-masked]> wrote:
    > Well, one change that would be that drastic would be changing it from a
    > format in which each block type has its own name space for options to a
    > format in which there's a single global name space for options, and
    > thus changing the option numbers.  That change would not only mean that
    > old code would no longer be able to read the new files, it would also
    > mean that code that handles the new files would need a different code
    > path to handle the old files.

Well, I don't feel too strongly about changing this.
I would suggest that we could leave values < 256 as "local", and provide new
numbers which were globally unique as aliases.  But that still leaves code
that is needed to read old files which likely never goes away, and so what's
the point.

    >> So I think that we should simply turn this into a 4-byte int32 field.
    >> Let's just put the highest *RFC* number that must be understood in
    >> order to comprehend all the blocks present.  At approx. 3500 RFCs/10
    >> years, 32-bits will last a long long long time... We could even do
    >> this with just the Minor Version and be good until rfc65535, which
    >> might be by oh year 2265 assuming the IETF continues at it's present
    >> rate.

    > So are you suggesting that updates to the format, such as the addition
    > of new blocks, new options, new link-layer types, etc. would be done
    > with RFCs?

To the extent that they would be "standards", yes.
If someone wants to allocate a block type for their own use, that would be
fine, it would just be RFCXXXX + extension.

    >> On the Interface Description Block, (section: 4.2), the table shown
    >> should be an IANA controlled registry.

    > I.e., replace http://www.tcpdump.org/linktypes.html with an IANA

Yes, I think that this is reasonable.

    >> The reference to section 3.5 is initially confusing when reading.  I
    >> think that there should be a single table (in section 3.5).  I see
    >> that the Section Header Block also reuses codes used in the IDB: I
    >> think that this is a mistake, the codes in a comment block should not
    >> be context dependant.  My pull request takes care of all of them, I
    >> hope.

    > Does your pull request also bump the current major version number?

Fair point, it should do that too if we were to accept this change.

    > I don't see what the problem is with different block types having
    > different option name spaces.  Common code to handle options in all
    > block types wouldn't be that useful - that code wouldn't, for example,
    > be able to do anything useful with the interface speed option in a Name
    > Resolution Block.  (And what do you mean by a "comment block"; we don't
    > have blocks whose sole purpose is to contain comments, we have a
    > comment option that can be used in any block, the content of which is
    > solely intended for display to humans.)

Perhaps I've mis-understood how these suboptions work.

    >> On the subject of the Encryption Block.  In discussion on the list
    >> many asked if we needed it.  Let me suggest that we do, but that yes,
    >> we need more information in it.

    > Well, the first question is what sort of information would be put in
    > it.  The current spec just says it "is used to store encrypted data"
    > and says of the data in it that "Once decrypted, it originates other
    > blocks.", but it doesn't say what sort of stuff is being encrypted!
    > Lists of credit card numbers? :-)

I think that essentially, it encrypts other blocks, such as captures, etc.

    >> maintain the table is not ideal.  Instead, let's create an IANA
    >> registry for it, populate it with existing values, and make the
    >> libpcap group the Expert Review.

    > So where would the actual formats be made available?

We'd have to include a few key ones into this document, and the rest would
continue to have the rather loose references that the web site has now.

    > I don't think an RFC is required as a reference, so we wouldn't have to
    > wait for an RFC to percolate through the process before making a new
    > link-layer header type available, but that would leave open the
    > question of what to use as a reference until the RFC comes out, and if
    > you don't want www.tcpdump.org to store the description until it shows
    > up in an RFC...

So, what happens right now is that we do Expert Review.  People ask us for a
link type, and we beat on them a bit to give a slightly stable document to
point at, or at least to argue why no existing type works for them.
Then they get the number, and if we are lucky, we also have a stable link to
their format.  With IANA Expert Review, we could essentially do exactly the
same thing --- just that IANA would write down the result.

Numbers can be assigned by RFC as well, and can be assigned early, and IANA
is happy to change the reference from draft-foo-bar to RFCxyaz as the
document progresses.
But, it can also just be a link to http://www.company.com/technote/foo.html,
or it can, as it says now, say, "Used by Company XYZ for Proprietary FOO"

---

From: Michael Richardson <[email-masked]>
Date: Thu, 12 Nov 2015 11:31:54 -0500

Warren Kumari <[email-masked]> wrote:
    > game to try progress it under OpsAWG.  W

On Sunday at the ETSI 6lo plugfest Dominique Barthel described to me a way
that the IOT-Lab is using *pcap* (with custom LINKTYPE_USR0...) in a
syslog/MILE/INCH-like way, except that they are debug events rather than
security events.  Interspersed with packets that were actually captured and
caused the debug... viewable in wireshark.

PCAP files already travel over TCP and SSH connections between collectors and
analyzers... so it's not just a format at rest anymore.

I am not subscribed to the opsawg list, I'm sure.

---

From: Guy Harris <[email-masked]>
Date: Fri, 13 Nov 2015 18:58:27 -0800

> Perhaps I've mis-understood how these suboptions work.

The idea is that each block type has its own set of options, with every set being required to include an option numbered 0 for "end of options" and an option numbered 1 for "UTF-8 comment string".  Code to read a particular block type handles all the options for that block type, including options 0 and 1; the code to read option 1 is simple, it's the code to *display* it that does most of the work, and that'd probably be different for different block types anyway.

> I think that essentially, it encrypts other blocks, such as captures, etc.

We should probably indicate that, then, rather than just vaguely stating "An Encryption Block is used to store encrypted data." and "Encrypted Data: data of this block. Once decrypted, it originates other blocks."

Or perhaps we should remove it from the spec, and put something back in once it's finalized to the point of having a block number assigned and having an indication of what it contains and how it should be handled by readers.

> continue to have the rather loose references that the web site has now.

The Web site also stores a few specs linked to from entries in the table at

	http://www.tcpdump.org/linktypes.html

for formats for which there's no spec to point to elsewhere on the Web; where should those specs be stored?

In addition, the description column in that table has some fairly long entries, which look longer than the sort of things that show up in IANA registries such as

That table just points to RFCs, and the main RFC:

	http://tools.ietf.org/html/rfc1761

just has a rather terse table:

        Datalink Type:

                A 32-bit (4 octet) field identifying the type of
                datalink header used in the packet records that follow.
                The datalink type codes are listed in the table below:

                Datalink Type           Code
                -------------           ----
                IEEE 802.3              0
                IEEE 802.4 Token Bus    1
                IEEE 802.5 Token Ring   2
                IEEE 802.6 Metro Net    3
                Ethernet                4
                HDLC                    5
                Character Synchronous   6
                IBM Channel-to-Channel  7
                FDDI                    8
                Other                   9
                Unassigned              10 - 4294967295

most of the entries of which are probably sufficient (i.e., they just contain a frame for the particular LAN/MAN technology, with nothing added, subtracted, or rearranged), but "HDLC", "Character Synchronous", and "IBM Channel-to-Channel" are too vague to actually be useful.  That might be because those DLPI types were reserved for those purposes but never actually officially *used* for those purposes, and thus nobody had to figure out what the format of those packets were.

We could publish an RFC containing the current set of link-layer header types from the tcpdump.org site, perhaps including the text from the specs stored on tcpdump.org, and refer to that from the pcap-ng spec *and* from any pcap spec we publish as an I-D or RFC; I'd have it give the descriptions from the tcpdump.org site, to be a little more restrictive than the snoop spec - I'd rather not have to throw heuristics into tcpdump or Wireshark to distinguish between packets using one interpretation of the description of LINKTYPE_FOOBAR and packets using another interpretation.

> is happy to change the reference from draft-foo-bar to RFCxyaz as the

So it sounds as if we can assign a number before the RFC is issued for a new block type/link type/option code/etc., and update the registry when an I-D is issued or updated or an RFC is issued.

> or it can, as it says now, say, "Used by Company XYZ for Proprietary FOO"

So presumably somebody who wants a link type would go to

	http://www.iana.org/form/protocol-assignment

and request one.

So does a Designated Expert for a registry get to edit whatever underlying file or database is used to generate pages for the registry in question?  I.e., what would the Designated Expert do when adding a new block type/link type/option/etc.?  Is there a repository to which they're granted access and to which they do a commit/push/whatever with the update?  Is it a database for which there's some Web interface to publish updates?

Or does the Designated Expert send a request to somebody at the IANA, who updates the file/database/whatever?

I assume the actual underlying mechanism that implements the Protocol Assignment registry pages is documented *somewhere*.