Subject: [DNSOP] New draft on representing DNS messages in JSON

From: Paul Hoffman <[email-masked]>
Date: Wed, 20 Aug 2014 10:22:26 -0700

Greetings again. I have a need for expressing DNS messages in JSON for an application I have written. Stephane and I had started to work on this but ended up dropping it. draft-dulaunoy-kaplan-passive-dns-cof comes at the same problem from another direction, but it is specific for one problem space and isn't using real JSON.

So, I whipped this one up. I intend it to be Experimental because I have no idea if anyone other than me wants it. Also, I don't think it should be a WG document if only a few folks have need for the format. However, I would love to heard comments and suggestions.

--Paul Hoffman

A new version of I-D, draft-hoffman-dns-in-json-00.txt
has been successfully submitted by Paul Hoffman and posted to the
IETF repository.

Name:		draft-hoffman-dns-in-json
Revision:	00
Title:		Representing DNS Messages in JSON
Document date:	2014-08-20
Group:		Individual Submission
Pages:		10
URL:            http://www.ietf.org/internet-drafts/draft-hoffman-dns-in-json-00.txt
Status:         https://datatracker.ietf.org/doc/draft-hoffman-dns-in-json/
Htmlized:       http://tools.ietf.org/html/draft-hoffman-dns-in-json-00

Abstract:
  Some applications use DNS messages, or parts of DNS messages, as
  data.  For example, a system that captures DNS queries and responses
  might want to be able to easily search those without having to decode
  the messages each time.  Another example is a system that puts
  together DNS queries and responses from message parts.  This document
  fully describes a standardized format for DNS message data in JSON.

---

From: Shane Kerr <[email-masked]>
Date: Thu, 21 Aug 2014 10:22:32 +0200

Paul,

On Wed, 20 Aug 2014 10:22:26 -0700
Paul Hoffman <[email-masked]> wrote:

> Greetings again. I have a need for expressing DNS messages in JSON
> for an application I have written. Stephane and I had started to work
> on this but ended up dropping it.
> draft-dulaunoy-kaplan-passive-dns-cof comes at the same problem from
> another direction, but it is specific for one problem space and isn't
> using real JSON.
> 
> So, I whipped this one up. I intend it to be Experimental because I
> have no idea if anyone other than me wants it. Also, I don't think it
> should be a WG document if only a few folks have need for the format.
> However, I would love to heard comments and suggestions.

Interesting. Some comments...

* I don't like the treatment of QNAME*/hostQNAME, NAME*/hostNAME, and
  so on. Since JSON includes encoded strings, wouldn't it make more
  sense just to always put the QNAME in there? (Especially since you'll
  end up with SRV queries always being encoded as they have underscore
  characters...)

* In general I'm not super enthusiastic about the mixing of binary and
  formatted data - I tend to think an application will want one or the
  other. Perhaps it makes more sense to define two formats, one binary
  and one formatted? Or...

* Maybe it makes sense to define a meta-record so consumers can know
  what to expect? Something that lists which names will (or may) appear.

I'd be mildly curious to see a comparison of the compressed sizes of
JSON-formatted data (without data duplicated as binary stuff) versus
non-JSON-formatted data. My intuition is that compression will remove
most of the horrible redundancy that is involved in JSON, but there's
only one way to be sure. ;)

Cheers,

--
Shane

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 07:01:01 -0700

Thanks for the comments!

>  end up with SRV queries always being encoded as they have underscore

JSON requires its strings to be encoded in a particular character set. Given that the labels in a QNAME/NAME can be an binary cruft, you can't assume that every QNAME will be representable.

I could reverse the logic and say QNAME/NAME are hostnames, and thus plain JSON strings in ASCII (which is by far the most common thing), and then have a different field for nonHostQNAME* and nonHostNAME* for when the QNAME/NAME. However, there would then need to be a way to indicate that the QNAME/NAME was not representable as a hostname. "" is OK, but then we would have to say that the root would always have to be represented as ".", which is not true now. To avoid that, I went with QNAME/NAME being as 1035 defined them, but I'm willing to think about other possibilities. 

>  other. Perhaps it makes more sense to define two formats, one binary

All fields are optional, so a profile could say "don't include these" or "always include those". Further, and more importantly, most RDATA are binary. I did not want to force implementations to use the presentation format for RDATA.

>  what to expect? Something that lists which names will (or may) appear.

That would be a JSON schema. Just using that phrase will cause screaming in the Apps Area. Having said that, it's perfectly reasonable for a profile to insist that each record have a profile indicator such as "Profile": "Private DNS interchange v3.1".

> only one way to be sure. ;)

Sure. It's pretty trivial to do, for example, a CBOR format that follows this; there are now CBOR libraries for most popular modern languages (see http://cbor.io/). If folks here want that, I can add it as an appendix. To be clear, however, I haven't heard anyone saying they want compression so badly they are willing to lose readability of the data.

--Paul Hoffman

---

From: Andreas Gustafsson <[email-masked]>
Date: Thu, 21 Aug 2014 18:18:16 +0300

Paul Hoffman wrote:
> JSON requires its strings to be encoded in a particular character
> set. Given that the labels in a QNAME/NAME can be an binary cruft,
> you can't assume that every QNAME will be representable.

There already exists a standard mechanism for representing this binary
cruft in the JSON-safe subset of ASCII, namely the \DDD notation of
RFC1035 section 5.1.  If you simply require the use of this notation
for any non-printable characters in a label, the resulting string will
be representable in JSON, and there is no need for separate NAME and
hostNAME encodings.

One potentially confusing aspect of this is that the backslashes will
then be subject to a second level of escaping by JSON.  But there's no
escaping (pun not intended) the fact that we are doing two layers of
translation: first from wire format binary data to human-readable
dotted names, and then from human-readable names to JSON syntax.

An implementation that uses existing, correctly implemented libraries
for each layer of translation will automatically do the right thing:
the DNS library will translate the length-prefixed wire format strings
to dot-delimited ones *and* do \DDD escaping, and the JSON library
will add a set quotes around the string to make it a JSON string *and*
double any backslashes.
-- 
Andreas Gustafsson, [email-masked]

---

From: Shane Kerr <[email-masked]>
Date: Thu, 21 Aug 2014 17:27:57 +0200

Paul,

On Thu, 21 Aug 2014 07:01:01 -0700
Paul Hoffman <[email-masked]> wrote:

> > * I don't like the treatment of QNAME*/hostQNAME, NAME*/hostNAME,
> > and so on. Since JSON includes encoded strings, wouldn't it make
> > more sense just to always put the QNAME in there? (Especially since
> > you'll end up with SRV queries always being encoded as they have
> > underscore characters...)
> 

I think you're making it too hard. Control characters, ", and \ are
already required to be escaped. Just specify a similar requirement for
octets 127 to 255 also be escaped, and we're done.

> > * In general I'm not super enthusiastic about the mixing of binary
> > and formatted data - I tend to think an application will want one
> > or the other. Perhaps it makes more sense to define two formats,
> > one binary and one formatted? Or...
> 
> All fields are optional, so a profile could say "don't include these"
> or "always include those". Further, and more importantly, most RDATA
> are binary. I did not want to force implementations to use the
> presentation format for RDATA.

The problem with an "all fields are optional" approach is that it puts
all the burden on the consumer of the data, right? You literally have
no idea what to expect. (That's kind of why I proposed some sort of
schema below.)

I understand not wanting to force implementations to use the
presentation format for RDATA... OTOH it seems likely that the reason
people are putting data in JSON is so they can see what it is. We could
always try the RFC 3597 approach for an unknown RTYPE?

> That would be a JSON schema. Just using that phrase will cause
> screaming in the Apps Area. Having said that, it's perfectly
> reasonable for a profile to insist that each record have a profile
> indicator such as "Profile": "Private DNS interchange v3.1".

Screaming aside, applications will either have an implicit schema or an
explicit one. Defining the problem to be out of scope may be necessary
to get something published, but that's a symptom of IETF brokenness
IMHO, since it reduces the usefulness of any such RFC. :(
 
> > non-JSON-formatted data. My intuition is that compression will
> > remove most of the horrible redundancy that is involved in JSON,
> > but there's only one way to be sure. ;)
> 
> Sure. It's pretty trivial to do, for example, a CBOR format that
> follows this; there are now CBOR libraries for most popular modern
> languages (see http://cbor.io/). If folks here want that, I can add
> it as an appendix. To be clear, however, I haven't heard anyone
> saying they want compression so badly they are willing to lose
> readability of the data.

Oh, I meant with gzip or the like, not some JSON crafted format.

So the idea is:

   $ tcpdump -w somefile.pcap
   $ pcap2dnsjson somefile.pcap somefile.json
   $ gzip somefile.pcap
   $ gzip somefile.json
   $ ls -l somefile.{pcap,json}.gz
   
Then compare the sizes of the compressed files.

The idea being that when moving files around via scp or rsync or
whatever they'd probably be compressed like this, and probably also for
medium-term storage. My hope is that a compressed JSON is roughly the
same size as a compress raw pcap file, since basically they have the
same entropy.

The reason I bring this up is to give a feel for the size cost of a
bloated text format in practice. :)

Cheers,

--
Shane

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 09:02:10 -0700

> octets 127 to 255 also be escaped, and we're done.

Andreas' and Shane's requests differ. And they both ignore the fact that JSON defines strings as Unicode characters, not as octets. The escaping "defined" in RFC 1035 does not say where it must be applied.

Note that the definition of a string in JSON is:

      string = quotation-mark *char quotation-mark

      char = unescaped /
          escape (
              %x22 /          ; "    quotation mark  U+0022
              %x5C /          ; \    reverse solidus U+005C
              %x2F /          ; /    solidus         U+002F
              %x62 /          ; b    backspace       U+0008
              %x66 /          ; f    form feed       U+000C
              %x6E /          ; n    line feed       U+000A
              %x72 /          ; r    carriage return U+000D
              %x74 /          ; t    tab             U+0009
              %x75 4HEXDIG )  ; uXXXX                U+XXXX

      escape = %x5C              ; \

      quotation-mark = %x22      ; "

      unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

Given this, and the likelihood that escaping is going to screw up NAME/QNAME exactly where it will be needed the most (to get the exact octets of an odd name), I think making NAME/QNAME only hold hostnames, and non-hostnames must be in a different field that is hex-encoded, will be the easiest to get right.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 09:11:18 -0700

> no idea what to expect. (That's kind of why I proposed some sort of

Correct.

> always try the RFC 3597 approach for an unknown RTYPE?

Define "unknown". Note that at least one "known" RTYPE has its description wrong, but you'd only know that if you read the RFC errata.

I could have RDATA is a string that is in presentation format, and hope that all emitters and receivers get that right, with an optional additional filed of rdataOctets! for a base64url-encoded version. If the emitter doesn't know the presentation form, it uses that only; if the emitter isn't sure that the receiver knows the presentation form, the emitter adds an rdataOctets! field as well. If the emitter knows that the RDATA is broken (such as a three-octet value for an A record), it just uses an rdataOctets! field. Would that suffice for you? 

> IMHO, since it reduces the usefulness of any such RFC. :(

The IETF looked at the horrid mess crated by XML schema, and decided that implicit schema described in text are better. And before you attack the messenger on this, note that (a) I tried to get the IETF to look at JSON schema again and (b) I'm the co-chair of the JSON WG.

> bloated text format in practice. :)

Noted. That's an easy thing for implementers of this format to do. So is saying that size is so important (maybe because we finally have so many DNSSEC keys passed about) that CBOR or something like it would be better.

--Paul Hoffman

---

From: Jay Daley <[email-masked]>
Date: Fri, 22 Aug 2014 07:53:53 +1200

I agree with all of Shane's points below, namely better encoding of data, not making all fields optional, handling of unknown rrtypes and having a schema for the data.  These are all points (and many more) that were thoroughly addressed in the following draft on representing DNS in XML

	http://tools.ietf.org/html/draft-daley-dnsxml-00

If it helps then I can produce XSLT that takes data in the XML format specified in that draft and validated against that schema and turn it into whatever flavour of JSON you require:

	http://controlfreak.net/xml-to-json-in-xslt-a-toolkit/

*stands back and waits for the "but json is sooo much easier than xml" brickbats*

Jay

On 22/08/2014, at 3:27 am, Shane Kerr <[email-masked]> wrote:

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

-- 
Jay Daley
Chief Executive
.nz Registry Services (New Zealand Domain Name Registry Limited)
desk: +64 4 931 6977
mobile: +64 21 678840
linkedin: www.linkedin.com/in/jaydaley

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 13:11:38 -0700

On Aug 21, 2014, at 12:53 PM, Jay Daley <[email-masked]> wrote:

> I agree with all of Shane's points below, namely better encoding of data,

I have responded to this one, assuming you mean RR data.

> not making all fields optional,

I don't think Shane made that argument. To me, it sounded like Shane was fine with fields being optional as long as the records could be defined by a schema.

> handling of unknown rrtypes

Sure, but that's already in the document.

> and having a schema for the data.

He preferred an internal schema; I explained why we couldn't do that with current or soon-to-be-expected standards.

>  These are all points (and many more) that were thoroughly addressed in the following draft on representing DNS in XML
> 

Not to be too picky about it, but your document doesn't describe "DNS in XML", it describes "DNS resource records in XML". You clearly say in section 2.5 that you don't try to cover the header data, which is very important for the use cases for this document.

> *stands back and waits for the "but json is sooo much easier than xml" brickbats*

There is that. In fact, one could argue that it would be much simpler to cast your document in JSON, if you didn't need a schema. Nothing is stopping people from adopting yours, although I would have called yours as experimental as mine.

--Paul Hoffman

---

From: Shane Kerr <[email-masked]>
Date: Fri, 22 Aug 2014 14:16:23 +0200

Paul,

On Thu, 21 Aug 2014 09:02:10 -0700
Paul Hoffman <[email-masked]> wrote: 
> Andreas' and Shane's requests differ. And they both ignore the fact
> that JSON defines strings as Unicode characters, not as octets. The
> escaping "defined" in RFC 1035 does not say where it must be applied.

While we propose different solutions, the thing that you think we're
ignoring is what we're actually just working around.

Both Andreas' and my suggestions work by recognizing that ASCII is a
subset of Unicode, and requiring that DNS JSON messages use ASCII.

>       unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

Yes, yes.

> Given this, and the likelihood that escaping is going to screw up
> NAME/QNAME exactly where it will be needed the most (to get the exact
> octets of an odd name), I think making NAME/QNAME only hold
> hostnames, and non-hostnames must be in a different field that is
> hex-encoded, will be the easiest to get right.

In Andreas' proposal a name of ^C would be written as "\\003", and in
mine it would be written as "\u0003". I'm not sure why you
think this would cause more more of a chance of error than "03".

To be honest, I think a more likely scenario is that a coder consuming
this data would not bother to look at any specifications, but build a
quick parser, which would then break every couple days as some random
packet has a "QNAME" instead of "hostQNAME" value show up. :)

Further, Andreas' and my proposal both have the nice property that an
SRV lookup would appear like "_sip._tcp.example.com", instead of 
"735f70695f2e63742e7078656d616c702e656f636d".

One slight advantage of my proposal over Andreas' is that a consumer
would likely not have to do anything fancy to read the data as it was
in the original message. (A producer might have to do some gymnastics
to insure that %x7F to %xFF are output properly, depending on how the
messages are generated, I suppose.)

To sum up, I think that if you're going to the bother of transforming
DNS messages into some vaguely human-readable format, you should try to
make it as readable as possible.

Cheers,

--
Shane

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 22 Aug 2014 08:49:47 -0700

> ignoring is what we're actually just working around.

Yes, exactly. :-)

> subset of Unicode, and requiring that DNS JSON messages use ASCII.

Do you mean any part of messages, or only names? I hope you would not encode a DNSKEY with either form of escaping you used.

> think this would cause more more of a chance of error than "03".

You picked the easy example. Just to be clear, for "copyright sign", Unicode U+00A9, you would encode this as "\u00a9", and Andreas would do so as \\169, yes? And that you are only encoding single octets, not Unicode characters? If so, then Andreas' proposal is much better than yours, because a label that actually contains non-ASCII characters encoded in UTF-8 (which we still unfortunately see plenty of) will most likely get encoded wrong in your scheme but correctly in Andreas'.

> packet has a "QNAME" instead of "hostQNAME" value show up. :)

Quite possibly true.

> "735f70695f2e63742e7078656d616c702e656f636d".

Yes, unless we take my proposal from yesterday, which was to make QNAME/NAME be the name unless it is not a hostname, in which case there is a second field.

> messages are generated, I suppose.)

I think yours is more fragile in the face of Unicode. For yours, you have to assure that the producer is not using any Unicode encoding scheme; Andreas' treats everything as octets more cleanly.

> make it as readable as possible.

Fully agree. But I also want to have a second format where it is clearly binary in case the producer screws up the escaping. I'm happy to make the QNAME/NAME be the easier-to-read one.

--Paul Hoffman