Subject: [Qlog] qlog in pcap-ng format

From: Michael Richardson <[email-masked]>
Date: Sat, 08 May 2021 15:42:58 -0400

I saw part of this presentation a few weeks ago someplace.
I just watched the SAAG recording.

1) I think that JSON is too verbose for this stuff, but I think that CBOR
   (described by CDDL) is probably exactly right for what you are doing.

2) PCAP-NG, which wireshark reads/writes, and libpcap only reads,
   accomodates a multitude of different streams and formats, not just
   ethernet dumps.
   (We have USB captures, nflog captures, and stuff all sorts of
   instruments).

PCAP and PCAP-NG are candidates for adoption by the OPSAWG, but if
QUIC would like to adopt them, that's okay with me.

I think that there is a significant value here, particular when there may be
a need to capture both TCP, UDP and internal logic at the same time.
Imagine something doesn't work, because something is eating UDP packets with
a particular checksum (or ethernet CRC).  It's happened more than once.
Having the internal state that says, "I retransmitted foo", and then the
capture from end A, that shows the encrypted QUIC packet going out, and
then the capture from end B, which says, "I didn't get foo, I asked for it
again"
would be very useful to see.  That requires merging streams from both ends,
and interleaving actual captured traffic at the same time.
(And still doesn't require access to cleartext to debug)

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Robin MARX <[email-masked]>
Date: Mon, 10 May 2021 10:35:16 +0200

Hello Michael,

Thank you for your comments.

1)
The qlog drafts are set to be adopted by the QUIC wg soon and this is
indeed still an open point of discussion.
This is tracked at https://github.com/quiclog/internet-drafts/issues/144
with already some thoughts on the matter mentioned there.

2)
I agree there is significant overlap between PCAP and qlog conceptually
(though I did not know pcap was actually considered for adoption at the
IETF).
However, from my understanding, the PCAP format is strongly oriented around
logging (full) raw packet contents with little support for omission of
fields / anonymization besides rough, high-level truncation (-s, snaplen?)
As such, while it would be possible to support
*encrypted*-quic-with-extra-comments,
it would be much more difficult to support *decrypted*-quic-with-extra-comments
taking into account privacy/file size/ease of capture/...
(I'm not even sure pcap allows the logging of decrypted protocol data
as-is? And even if it does, is it still compatible with e.g., wireshark or
other pcap tools?)

Additionally, again from my understanding, pcap is a custom binary format
that's not all that easy to process/use.
IIUC, even tools like pyshark utilize tshark's XML output instead of
parsing the PCAP itself  (which has been a source of much frustration on my
end...).
This seems to go somewhat against one of the main goals of qlog, which is
to make the creation of tools and the analysis of data easier (which is one
of the main reasons to choose JSON in the first place).

That being said, I'm far from an expert on PCAP/PCAP-NG and could be wrong
on all these counts. Please feel free to educate me.
However, if I'm not, I feel it's still best to continue qlog as a separate,
complementary approach to pcaps.

Finally, I've added the QUIC wg in CC, as that's where most of the
work/discussion will likely be done in the future.

With best regards,
Robin

On Sat, 8 May 2021 at 21:43, Michael Richardson <[email-masked]>
wrote:

> Qlog mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/qlog
>

-- 

dr. Robin Marx
Postdoc researcher - Web protocols
Expertise centre for Digital Media

*Cellphone *+32(0)497 72 86 94

www.uhasselt.be
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

---

From: Michael Richardson <[email-masked]>
Date: Mon, 10 May 2021 10:45:06 -0400

> (though I did not know pcap was actually considered for adoption at the

Please note that pcap (legacy) is intended as Informational.
PCAP-NG (I hate "NG" for a format that is now 10 years old!), is intended as
Standards Track.

    > However, from my understanding, the PCAP format is strongly oriented

Your understanding is accurate for PCAP, but not for PCAP-NG.
If I had a do-over, PCAP-NG would have been pure-CBOR.

    > work/discussion will likely be done in the future.

Then, I suggest maybe the qlog ML should get killed, if you want discussion
on the QUIC list.

---

From: Robin MARX <[email-masked]>
Date: Wed, 19 May 2021 13:49:07 +0200

Hello Michael,

Sorry for the late reply, but gmail decided your mail belonged in my spam
folder for some unfathomable reason.

I must then admit my lack of knowledge on the PCAP-NG format and its
possibilities and support in e.g., open source libraries.
I've added a comment/TODO for this in the issue where we're tracking the
qlog serialization format at
https://github.com/quiclog/internet-drafts/issues/144#issuecomment-844023310
Any initial materials you might provide for this would be most welcome.

The question about what to do with the separate qlog mailing list now that
we're eyeing QUIC wg adoption will be escalated to the ADs. Thanks for
mentioning this.

With best regards,
Robin

-- 

dr. Robin Marx

www.uhasselt.be