Subject: Re: [Qlog] [OPSAWG] adopting draft-gharris-opsawg-pcap and draft-tuexen-opsawg-pcapng

From: Michael Richardson <[email-masked]>
Date: Sun, 04 Jul 2021 21:18:59 -0400

Guy Harris <[email-masked]> wrote:
    >> The proposal is to adopt draft-gharris-opsawg-pcap as an Informational
    >> document.  This documents the ~30 year old pcap format used by tcpdump,
    >> wireshark, etc.   Almost all of IPv6, DNSSEC, DNS extensions, etc. research
    >> done by many researchers, including for instance, the
    >> https://www.caida.org/projects/network_telescope/ have used pcap files as
    >> their capture format.
    >> We need to do this as *WG* and can not do this as ISE, because the pcap
    >> document establishes the critical LinkType registry.  One of the exchanges
    >> above is about how to "load" this rather large legacy into IANA.

    > Do the link-layer types belong in a pcap I-D/RFC, given that they're
    > used in both pcap and pcapng, and given that there are a lot of them,
    > so that the link-layer types information might swap the pcap format
    > information if they're in a single spec?

Some document which would be the result of IETF Consensus needs to create the
link-layer repository, and it needs to initialize the database.

I think that you mean "swamp" where you write "swap"?
I don't think it's a problem that they are in the same document.
The WG could disagree, and tell us to split the document into two.

    > We might want to have a pcap spec, a pcapng spec, and another document
    > that gives detailed descriptions - as detailed as what's in
    > https://www.tcpdump.org/linktypes.html

The IANA Considerations rules for the repository is "Specification Required",
but that doesn't mean that the specification has to be in the document we
submit.  It can be any stable document.  That includes an RFC, but isn't
restricted to that.
It is perfectly fine to point at linktypes.html from the IANA Registry.
There are presently more details in the table than I originally had intended
to include.

    >> pcapng would be an IETF controlled document, the "pcap 2.0", but we can't
    >> really do as many changes as we might like.
    >> (I'd sure like to name it pcap2.0, as I hate the whole "Next Generation"
    >> moniker, but I don't know if that would fly at this late stage).
    >>
    >> *MAYBE* PCAPNG should also be Informational, given that we can't really mess
    >> with it too much.
    >>
    >> *MAYBE* PCAP2.0 should be just the structure as IETF Standards Track, with
    >> the section structure which is in PCAPNG (which is not changeable) should be
    >> an Informational document.  This involves more documents, but no additional
    >> text.

    > So, with that proposal for pcapng/pcap 2.0 would there be, as separate documents:

    > the overall structure, currently covered by sections 1-4 and 6-7 of the
    > spec, and not very changeable;

Yes.  Call this "pcap2"

    > the definitions of blocks and block-specific options, currently covered
    > by sections 4 and 5, with the blocks currently specified not being
    > changeable other than using reserved fields and adding new options, but
    > with adding new block types being allowed;

I would include, in "pcap2" all the current blocks that correspond to capture
types present in pcap1.
Then, I'd put all the things which aren't about capture of IP packets
(e.g. systemd, USB, ...) into another (informational) document.
Or perhaps just leave them in a web page, having pointed at them in pcap2's
IANA Considerations.

    > and a registry for block and block-specific option types, with the
    > registry pointing to the second document or to any new documents
    > specifying new block types?

Well, we can't make forward pointers to documents that don't exist yet.
So, the forward pointers wind up in the IANA registry.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Guy Harris <[email-masked]>
Date: Sun, 4 Jul 2021 16:49:59 -0700

On Jun 29, 2021, at 5:29 PM, Michael Richardson <[email-masked]> wrote:

> There was many comments about how we should have used CBOR for PCAPNG, and I
> would agree that in hindsight, it would be good.  New section types for
> PCAPNG could well be encoded in CBOR.  I also advocated that for
> draft-ietf-quic-qlog-h3-events,draft-ietf-quic-qlog-main-schema,
> draft-ietf-quic-qlog-quic-events, but they didn't go that way. (At least, not
> yet?).  I haven't time to do QUIC stuff, but if there is interest, and I'll
> read the documents above, and see if I can come up with a proposal.
> 

Do the link-layer types belong in a pcap I-D/RFC, given that they're used in both pcap and pcapng, and given that there are a lot of them, so that the link-layer types information might swap the pcap format information if they're in a single spec?

We might want to have a pcap spec, a pcapng spec, and another document that gives detailed descriptions - as detailed as what's in

of the existing types.  The registry could point to the I-D/RFC for descriptions of the types; new types would get new RFCs.

	the overall structure, currently covered by sections 1-4 and 6-7 of the spec, and not very changeable;

	the definitions of blocks and block-specific options, currently covered by sections 4 and 5, with the blocks currently specified not being changeable other than using reserved fields and adding new options, but with adding new block types being allowed;

and a registry for block and block-specific option types, with the registry pointing to the second document or to any new documents specifying new block types?

---

From: Guy Harris <[email-masked]>
Date: Sun, 4 Jul 2021 18:39:27 -0700

> link-layer repository, and it needs to initialize the database.

That document doesn't have to be the same document that describes the structure of pcap files, just as there could be separate documents for the structure of pcapng/pcap 2.0 file and for the initial set of block types for pcapng/pcap 2.0.

> I think that you mean "swamp" where you write "swap"?

Yes.

> There are presently more details in the table than I originally had intended

Above, you say

> link-layer repository, and it needs to initialize the database.

Does "initialize the database" mean "provide the initial set of entries in the database"?  If so, does it have to provide anything more than "see the entry for LINKTYPE_XYZZY in www.tcpdump.org/linktypes.html for the XYZZY linktype"?  Or, with an extra layer of compression, just have a list of LINKTYPE_XYZZY values with a single "for each linktype in this list, see the entry for that Linotype in www.tcpdump.org/linktypes.html"?

> I would include, in "pcap2" all the current blocks that correspond to capture

("pcap1" presumably being the current pcap file format.)

What do you mean by "capture types"?

If you're referring to linktypes, different linktypes don't have different block types.  The only current (as opposed to deprecated) block types corresponding to what appears in pcap files are the Enhanced Packet Block (EPB) and the Simple Packet Block (SPB), and the latter is arguably different from a pcap record given that it doesn't have a time stamp - "editcap -T pcap pcap-file.pcap pcapng-file.pcapng" will write out one Section Header Block, one Interface Description Block, and a pile of EPBs.

> (e.g. systemd, USB, ...) into another (informational) document.

systemd journal entries and USB packets are different here - USB packets are stored in EPBs, just as they're stored in pcap files (i.e., USB packets have a LINKTYPE_ value usable in both pcap files and pcapng Interface Description Blocks - well, several of them, given that different OSes supply different pseudo-headers), but systemd journal entries have their own block type.

> Or perhaps just leave them in a web page, having pointed at them in pcap2's

So have the bulk of the information about linktypes, block types, and block-specific option types be on some web site not in in ietf.org, with the registry just giving linktype/block type/option names and numbers, along with a link to that site?