Subject: [DNSOP] I-D Action: draft-ietf-dnsop-dns-capture-format-05.txt

From: [email-masked]
Date: Thu, 22 Feb 2018 04:27:48 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Domain Name System Operations WG of the IETF.

        Title           : C-DNS: A DNS Packet Capture Format
        Authors         : John Dickinson
                          Jim Hague
                          Sara Dickinson
                          Terry Manderson
                          John Bond
	Filename        : draft-ietf-dnsop-dns-capture-format-05.txt
	Pages           : 62
	Date            : 2018-02-22

Abstract:
   This document describes a data representation for collections of DNS
   messages.  The format is designed for efficient storage and
   transmission of large packet captures of DNS traffic; it attempts to
   minimize the size of such packet capture files but retain the full
   DNS message contents along with the most useful transport metadata.
   It is intended to assist with the development of DNS traffic
   monitoring applications.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-dnsop-dns-capture-format/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-dnsop-dns-capture-format-05
https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-dns-capture-format-05

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-dnsop-dns-capture-format-05

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Sara Dickinson <[email-masked]>
Date: Thu, 22 Feb 2018 12:31:29 +0000

Hi Folks, 

We have an update to draft which we hope captures all the comments to-date. 

- Make all data items in Q/R, QuerySignature and Malformed Message arrays optional
- Re-structure the FilePreamble and ConfigurationParameters into BlockParameters
- BlockParameters has separate Storage and Collection Parameters
- Storage Parameters includes information on what optional fields are present, and flags specifying anonymisation or sampling
- Addresses can now be stored as prefixes.
- Switch to using a variable sub-second timing granularity
- Add response bailiwick and query response type
- Add specifics of how to record malformed messages
- Add implementation guidance
- Improve terminology and naming consistency

There are still a number of ‘QUESTIONS’ in the draft that we would appreciate feedback on. 

Regards

Sara. 

> On 22 Feb 2018, at 12:27, [email-masked] wrote:
> 
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 23 Feb 2018 13:56:21 +0100

On Thu, Feb 22, 2018 at 12:31:29PM +0000,
 Sara Dickinson <[email-masked]> wrote 

> We have an update to draft which we hope captures all the comments

I think so, too. At least, it captured mine :-)

>   QUESTION: Should fields be added to indicate the sampling/
>   anonymisation method used?  If so, it is proposed to use a text
>   string and RECOMMEND it contain a URI pointing to a resource
>   describing the method used.

Yes. We cannot use a registry because the number of possible methods
is high, and changing, and each method has parameters. So, the idea of
the URL seems OK.

>   QUESTION: Should there be another flag to indicate that names have
>   been normalised (e.g. converted to uniform case)?

Yes. Otherwise, you cannot survey if draft-vixie-dnsext-dns0x20 is
used ot not.

>  QUESTION: No EDNS(0) option currently includes a name, however if one
>  were to include a name and permit name compression then both these
>  mechanisms would fail.

Is it even possible? RFC 6891 does not mention it. But it says
"OPTION-DATA. MUST be treated as a bit field." which seems to imply
that storing the entire OPT RR, or the option data, as a blob, is
perfectly OK.

>  QUESTION: There has been no feedback to date requesting further work
>  on the processing partially malformed messages.  The editors are
>  inclined not to include it in this version.  It could be the subject
>  of a future extension.

I agree. We waited for this document long enough. 

Editorial:

> Adherence to the second two rules given in Section 3.9 of [RFC7049

Isn't it clearer to say "the last two rules"? "The second two rules"
(meaning the third and the fourth) confused me.

---

From: Richard Gibson <[email-masked]>
Date: Wed, 28 Feb 2018 01:36:50 -0500

Thank you so much for the changes, I am confident that this format is 
now in a state where my organization could build on top of it to suit 
our needs. I have a handful of remaining comments below, mostly 
describing ways in which that process could be made more straightforward 
and standardized.

* Section 6.2.1 and 7.5.3.4: At minimum, RR field "ttl" should be 
optional in order to accommodate aggregation of resolver responses in 
which that field is decremented over time. But probably "rdata-index" 
should be optional as well.

* Section 7.1: Negative integer map keys are concise, but they come with 
the potential for interoperability-hostile collisions. I think the 
document should encourage restricting their use to tightly-controlled 
closed systems, and then additionally propose a particular namespacing 
pattern (e.g., "{namespaceName}-{keyName}") for use of 
implementation-specific /string/ keys in other cases. Further, it should 
explicitly reserve positive integer keys for future standardized 
extensions to the C-DNS format.

* Section 7.4.1: Why is BlockParameters an array instead of a map?

* Section 7.4.1.1: StorageParameters fields "opcodes" and "rrtypes" 
could be defined better... are they values /recorded/ (i.e., that were 
actually observed) or values that are /recordable/ (i.e., that the 
collection application was looking for)? And shouldn't they be optional 
(in either case, but *especially* the latter, lest implementations write 
out every possible rrtype value).

* Section 7.5.3: It's very good that the BlockTables "ip-address" items 
can be truncated by prefix length, but we have actually run into cases 
that call for /variable/ truncation (i.e., more precision in some 
subnets than in others), especially true given that a single table holds 
both client and server address data. What would you think about 
representing addresses not as direct byte strings, but rather as 
variable-length arrays or maps—index 0 encoding the prefix as a byte 
string, and optional index 1 (when present) encoding a prefix-length 
override? Or, if the extra size per address is too much, at least 
allowing such representations via polymorphism (byte string for default 
prefix-length, array/map for non-default)?

* Section 7.5.3.2: QueryResponseSignature field "qr-transport-flags" 
seems to reserve 1 bit /each/ for UDP, TCP, TLS, and DTLS, but it seems 
to me that UDP is mutually exclusive with TCP and that TLS is mutually 
exclusive with DTLS. Couldn't that data be represented in two bits (one 
in which 0 ⇒ UDP and 1 ⇒ TCP, the other in which 0 ⇒ plain-text and 1 ⇒ 
TLS)?

On 02/22/2018 07:31 AM, Sara Dickinson wrote:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Ddnsop-2Ddns-2Dcapture-2Dformat_&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=522TfZMUJHLQox2FWFORwOQXhrlqwp36X9Ty1v54YeI&e=
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Ddnsop-2Ddns-2Dcapture-2Dformat-2D05&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=Hc60T3mAF7VZgnUYR0LyhsTUTloEhCJIH3wIfUyCYmU&e=
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_html_draft-2Dietf-2Ddnsop-2Ddns-2Dcapture-2Dformat-2D05&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=oT9ecpv59bEHyd7Tpu69VbCmXsTBJYi3ePPp8kkuQGc&e=
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_rfcdiff-3Furl2-3Ddraft-2Dietf-2Ddnsop-2Ddns-2Dcapture-2Dformat-2D05&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=WAyfWMq1ADNBRPEAlbUe-vsBwe1E7XGilSC9g498WkM&e=
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.ietf.org_internet-2Ddrafts_&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=RS6uAkaKhYyD4CDe2_bX21qAvs0hkRPcFZ02B5wk_Kc&e=
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_dnsop&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=8dC7W8NvScBDakKiehjmI17kYXh39QFcg488yknExDo&e=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_dnsop&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-o8MJF7i0TzXAJRB0ncfTVfWKSyTG7nl_iTLU_A2B7c&m=S0pguE4aKgWT-7tOAVBInPr7s58myqhSmv9WtwMzeNA&s=8dC7W8NvScBDakKiehjmI17kYXh39QFcg488yknExDo&e=

---

From: Jim Hague <[email-masked]>
Date: Wed, 28 Feb 2018 15:56:11 +0000

> I think so, too. At least, it captured mine :-)

Good. And thanks for the feedback. We've noted your answers to our
'QUESTIONS' and other comments.

> that storing the entire OPT RR, or the option data, as a blob, is

I am also unclear as to whether storing a compressed name in an option
is permitted. As you say, the language seems to imply treating the data
as a blob is the only option, which suggests that it's not permitted.
We'd be interested in any contrary opinions.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Jim Hague <[email-masked]>
Date: Wed, 28 Feb 2018 17:21:55 +0000

On 28/02/2018 06:36, Richard Gibson wrote:
> our needs.

Thanks for the feedback. Much appreciated!
> should be optional as well.

We anticipated possibly doing this if anyone asked. We'll certainly go
back over that.

> implementation-specific /string/ keys in other cases.

We'll discuss this. We absolutely meant for negative keys to be
restricted to closed systems. Allowing string keys in addition is an
interesting idea; off the cuff, I wonder whether folk will want to pay
the file space penalty.

> Further, it should explicitly reserve positive integer keys for future standardized

That is absolutely the intention. We'll add some text making that clear.

> * Section 7.4.1: Why is BlockParameters an array instead of a map?

That's a typo. It should be a map, as it is in the CDDL.

> out every possible rrtype value).

The intention is that they should be /recordable/ values - in line with
the rest of the StorageParameters fields, they let a reader distinguish
values that aren't present because the writer didn't understand them and
values not present because they didn't occur in the data stream. And
yes, we're thinking implementations should explicitly write each opcode
and rrtype they understand. And possibly should treat opcodes and
rrtypes they don't understand as malformed. The thinking here is that
if, say, an rrtype the writer does not know how to decode contains a
compressed label, it's not going to be correctly handled, and we should
not be recording potentially known-broken data in the file as not malformed.

> prefix-length, array/map for non-default)?

Variable truncation is a use case I for one was not anticipating. We'll
discuss this. Allowing an address to be a bstr OR an array or map with
optional prefix sounds an interesting idea.

> in which 0 ⇒ UDP and 1 ⇒ TCP, the other in which 0 ⇒ plain-text and 1 ⇒

We need to clarify the language. qr-transport-flags is a single bit
IPv4/v6, a 4 bit number indicating the protocol, and a final single bit
indicating the presence of trailing data in the query. The protocol
values are (currently) UDP=0, TCP=1, TLS=2, DTLS=3. We did consider a
UDP/TCP bit and a !TLS/TLS bit, but felt this (a) might imply a closer
connection between TLS and DTLS than is the case, and (b) would not
easily extend to possible future schemes like DOH and DNS over QUIC. So
we went for a number with sufficient range to add another 12 options
before trouble happens. We considered breaking the number out to a
separate integer, but felt that this was likely, in practice, to be
unnecessary and so space efficiency considerations took precedence.
-- 
Jim Hague - [email-masked]          Never trust a computer you can't lift.

---

From: Richard Gibson <[email-masked]>
Date: Wed, 28 Feb 2018 22:52:13 -0500

> interesting idea; off the cuff, I wonder whether folk will want to pay
The cost of interoperability. An understandable concern for raw files, 
but repetitions of the same string keys would compress extremely well.
> and rrtype they understand.
Especially in our closed system, I don't see us keeping current another 
list of supported RR types. So we'd probably leave the array empty, 
violating the spirit of the spec if not the letter of it. But on the 
plus side, the size of the complete list in CBOR was smaller than I 
anticipated—less than 100 bytes.
> The thinking here is that
I sure hope that no new types violate RFC 3597 section 4. The problems 
with corrupted data in captures pale in comparison to the problems with 
corrupted data in live messages.
> unnecessary and so space efficiency considerations took precedence.
Thank you, that explanation greatly clears things up and the reasoning 
is sensible. I would recommend updating the text to something like "Bit 
1-4. Transport. 0000 = plain-text UDP, 0001 = plain-text TCP, 0010 = TLS 
[over TCP?], 0011 = DTLS [over UDP?]" (with a note in Section 7.2 that 
bit sequences are presented in descending significance).