Subject: [DNSOP] Character encoding of URI Target RDATA?

From: Robert Edmonds <[email-masked]>
Date: Mon, 15 Jun 2015 23:44:54 -0400

Hi,

According to RFC 7553 §4.5, the Target field in URI RDATA is a "sequence
of octets":

   The RDATA for a URI RR consists of a 2-octet Priority field, a
   2-octet Weight field, and a variable-length Target field.

   Priority and Weight are unsigned integers in network byte order.

   The remaining data in the RDATA contains the Target field.  The
   Target field contains the URI as a sequence of octets (without the
   enclosing double-quote characters used in the presentation format).

There doesn't seem to be any further information about how the sequence
of octets is encoded or decoded.  According to RFC 3986 §2,

   The URI syntax provides a method of encoding data, presumably for the
   sake of identifying a resource, as a sequence of characters.  The URI
   characters are, in turn, frequently encoded as octets for transport
   or presentation.  This specification does not mandate any particular
   character encoding for mapping between URI characters and the octets
   used to store or transmit those characters.  When a URI appears in a
   protocol element, the character encoding is defined by that protocol;
   without such a definition, a URI is assumed to be in the same
   character encoding as the surrounding text.

What character encoding should be used when decoding the Target field of
a URI RR?  It seems to me that the protocol transmitting the octets
hasn't defined the encoding of the octets (it just says that there's a
sequence of them) and that there's no "surrounding text" in a DNS wire
format message that can inform the decoder either.

Thanks!

-- 
Robert Edmonds

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 16 Jun 2015 14:53:56 +0900

Robert Edmonds wrote:

> a URI RR?

It depends on <host> part of URI, which decodes the URI.

How non-ASCII characters in zone files of a name server are
represented is a local issue to the name server.

						Masataka Ohta

---

From: Robert Edmonds <[email-masked]>
Date: Tue, 16 Jun 2015 12:30:49 -0400

> It depends on <host> part of URI, which decodes the URI.

No, I'm not talking about the encoding of components within the URI into
URI characters, I'm talking about the encoding of URI characters into
octets.  I.e., the second sentence of this paragraph, not the first:

   or presentation.

> represented is a local issue to the name server.

This is the *en*coding of characters in a zone file into wire data
octets.  (And, anyway, the zone file format is flexible enough that it
can load arbitrary octets, via \DDD escapes.)  How should a receiver
decode the wire data octets?

-- 
Robert Edmonds

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 17 Jun 2015 03:09:24 +0900

Robert Edmonds wrote:

> This is the *en*coding of characters in a zone file into wire
> data octets.

I'm afraid you are totally confused.

> How should a receiver decode the wire data octets?

Into a zone file? Or?

					Masataka Ohta

---

From: Robert Edmonds <[email-masked]>
Date: Tue, 16 Jun 2015 18:36:04 -0400

> I'm afraid you are totally confused.

Actually, I don't really see how zone files are relevant to my question.

> > How should a receiver decode the wire data octets?

The URI RR RFC says that URI RRs store octet sequences that represent
URIs, and says that URIs are specified in the URI RFC (3986).

The URI RFC defines URIs in terms of codepoints that are "based on the
US-ASCII coded character set".

What I'm asking is how the octet sequences provided by the URI RR RFC
are decoded into the sequences of URI characters used by the URI RFC.
Is there a generic way to do this, or does it depend on the specific
protocol (e.g., HTTP), or is it left up to the application?

-- 
Robert Edmonds

---

From: "John Levine" <[email-masked]>
Date: 16 Jun 2015 22:57:04 -0000

>protocol (e.g., HTTP), or is it left up to the application?

As far as I can see, RFC 3986 defines URIs as sequences of ASCII
characters.  In the few places where they mention non-ASCII material,
it says to represent them as percent encoded UTF-8, so it's still all
ASCII.

Can you give an example of URI RDATA where it would make sense to
interpret it other than as ASCII?

I suppose to be perfectly clear we might either say "percent encode
everything" or we might say "unencoded UTF-8 is allowed."  They're
both unambigious, and I expect most parsers can handle both.

R's,
John

---

From: Robert Edmonds <[email-masked]>
Date: Tue, 16 Jun 2015 21:45:23 -0400

> it says to represent them as percent encoded UTF-8, so it's still all

OK.  That RFC seems to distance itself from mere octets.

> interpret it other than as ASCII?

This is the FTP example from the URI RR RFC, to which the UTF-8 byte
order mark has been gratuitously added:

    TYPE256 \# 36 000a0001efbbbf6674703a2f2f667470312e6578616d706c652e636f6d2f7075626c6963

or, equivalently,

    URI 10 1 "\239\187\191ftp://ftp1.example.com/public"

Attempting to decode it as ASCII simply does the wrong thing, but I
don't see any reason that it's not a valid URI RR, and, knowing that
it's encoded as UTF-8 w/ BOM, it can be successfully parsed into a URI
(provided the Target field is handed off to the URI-parsing application
as raw bytes, and not as a string with DNS zone file \DDD style
escapes).

> both unambigious, and I expect most parsers can handle both.

It would be very nice indeed if application developers did not have to
guess at the encoding of the bytes.

-- 
Robert Edmonds

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 16 Jun 2015 23:06:41 -0300

> order mark has been gratuitously added:

Hmm...what RFC are you referring to? I can not find this in RFC 7553.

>  TYPE256 \# 36 
> 000a0001efbbbf6674703a2f2f667470312e6578616d706c652e636f6d2f7075626c6963
>
> or, equivalently,
>
> (provided the Target field is handed off to the URI-parsing 
> application as raw bytes, and not as a string with DNS zone file \DDD 
> style escapes).

The RFC says this:

This field holds the URI of the target, enclosed in double-quote
characters ('"'), where the URI is as specified in RFC 3986
[RFC3986].  Resolution of the URI is according to the definitions for
the Scheme of the URI.

> guess at the encoding of the bytes.

Earlier versions of the I-D did say explicitly that UTF-8 encoded 
characters is how the Target is to be interpreted, but feedback gave 
that it is better to just reuse the same specification as URIs. I.e. the 
interpretation is according to RFC 3986 (which implies unclear where 
3986 might be unclear).

    Patrik

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 17 Jun 2015 12:38:22 +0900

> Actually, I don't really see how zone files are relevant to my question.

Then, as the only case DNS perform character encoding/decoding
is when it read/write the zone files, you are asking a wrong
questions in a wrong ML.

> What I'm asking is how the octet sequences provided by the URI RR RFC

The RFC does not provide the octet sequences. Zone files do.

						Masataka Ohta

---

From: Tony Finch <[email-masked]>
Date: Wed, 17 Jun 2015 10:30:40 +0100

> protocol (e.g., HTTP), or is it left up to the application?

Are you after RFC 20 as opposed to IBM ASCII or something?

http://dotat.at/tmp/IBM-ASCII.png
http://www6.in.tum.de/pub/Main/TeachingWs2010Echtzeitsysteme/IBM360-Amdahl_april64.pdf

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Biscay: Variable, mainly northerly, 3 or 4, becoming northwesterly 5 in east.
Slight becoming moderate. Occasional drizzle later in north. Good,
occasionally poor later in north.

---

From: Robert Edmonds <[email-masked]>
Date: Wed, 17 Jun 2015 14:13:09 -0400

Patrik Fältström wrote:
> >This is the FTP example from the URI RR RFC, to which the UTF-8 byte order
> >mark has been gratuitously added:
> 

Sorry, it's not from an RFC.  I took one of the examples from RFC 7553,
and modified it to show a pathological example for John's question.

> Earlier versions of the I-D did say explicitly that UTF-8 encoded characters
> is how the Target is to be interpreted, but feedback gave that it is better
> to just reuse the same specification as URIs. I.e. the interpretation is
> according to RFC 3986 (which implies unclear where 3986 might be unclear).

I don't see anywhere in RFC 3986 where it says how to interpret an
arbitrary octet sequence as a URI.  In fact, it repeatedly emphasizes
that the sequence of characters forming a URI is decoupled from possible
encodings of that sequence into octets.

RFC 3986 §2:

    2.  Characters

       The ABNF notation defines its terminal values to be non-negative
       integers (codepoints) based on the US-ASCII coded character set
       [ASCII].  Because a URI is a sequence of characters, we must invert
       that relation in order to understand the URI syntax.  Therefore, the
       integer values used by the ABNF must be mapped back to their
       corresponding characters via US-ASCII in order to complete the syntax
       rules.

There are two encoding steps described here.  The first is the
production of a URI from its components into "URI characters", which
uses the percent-encoding scheme that everyone's familiar with to escape
URI components.  These "URI characters" are ABNF terminal values.  The
second encoding step is the conversion of these ABNF values into a
concrete octet stream.  Only this second encoding step is relevant for
the URI DNS RR, because serialized URIs have already undergone
%-encoding.

"Network ASCII" is a very common encoding for ABNF terminal values, but
not the only possible encoding.  RFC 5234 (ABNF):

    2.3.  Terminal Values

       Rules resolve into a string of terminal values, sometimes called
       characters.  In ABNF, a character is merely a non-negative integer.
       In certain contexts, a specific mapping (encoding) of values into a
       character set (such as ASCII) will be specified.

    [...]

    2.4.  External Encodings

       External representations of terminal value characters will vary
       according to constraints in the storage or transmission environment.
       Hence, the same ABNF-based grammar may have multiple external
       encodings, such as one for a 7-bit US-ASCII environment, another for
       a binary octet environment, and still a different one when 16-bit
       Unicode is used.  Encoding details are beyond the scope of ABNF,
       although Appendix B provides definitions for a 7-bit US-ASCII
       environment as has been common to much of the Internet.

       By separating external encoding from the syntax, it is intended that
       alternate encoding environments can be used for the same syntax.

    [...]

    Appendix B.  Core ABNF of ABNF

       This appendix contains some basic rules that are in common use.
       Basic rules are in uppercase.  Note that these rules are only valid
       for ABNF encoded in 7-bit ASCII or in characters sets that are a
       superset of 7-bit ASCII.

    [...]

    B.2.  Common Encoding

       Externally, data are represented as "network virtual ASCII" (namely,
       7-bit US-ASCII in an 8-bit field), with the high (8th) bit set to
       zero.  A string of values is in "network byte order", in which the
       higher-valued bytes are represented on the left-hand side and are
       sent over the network first.

RFC 3986 specifically declines to specify a particular concrete encoding
for URI characters:

    "This specification does not mandate any particular character
    encoding for mapping between URI characters and the octets used to
    store or transmit those characters."

and leaves it up to the protocol that embeds URIs to define the
encoding:

    "When a URI appears in a protocol element, the character encoding is
    defined by that protocol; without such a definition, a URI is
    assumed to be in the same character encoding as the surrounding
    text."

Unlike, say, an email message or webpage, there's no "surrounding text"
in a binary DNS response message that a consumer of URI RRs can rely on
to inform its choice of character encoding.

In fact, there are alternative encodings of URIs contemplated in an
example later in 3986:

    6.2.1.  Simple String Comparison

       [...]

       This character comparison requires that each pair of characters be
       put in comparable form.  For example, should one URI be stored in a
       byte array in EBCDIC encoding and the second in a Java String object
       (UTF-16), bit-for-bit comparisons applied naively will produce
       errors.  It is better to speak of equality on a character-for-
       character basis rather than on a byte-for-byte or bit-for-bit basis.
       In practical terms, character-by-character comparisons should be done
       codepoint-by-codepoint after conversion to a common character
       encoding.

       [...]

So, IMO, RFC 7553 is underspecified: it doesn't define an unambiguous
meaning for the octets appearing in a URI RR's Target field, so it's up
to applications to decide what character encoding to apply.  Was that
the intention?

-- 
Robert Edmonds

---

From: Robert Edmonds <[email-masked]>
Date: Wed, 17 Jun 2015 14:16:38 -0400

> Are you after RFC 20 as opposed to IBM ASCII or something?

Yes, it appears that's one possible encoding of URI characters :-)

-- 
Robert Edmonds

---

From: bert hubert <[email-masked]>
Date: Wed, 17 Jun 2015 21:52:52 +0200

> The RFC does not provide the octet sequences. Zone files do.

This is indeed correct. We can ignore what characters are in the URI and
just stuff them in the packet, it doesn't matter.

The question does remain how we encode them in a zone file. And indeed, this
is perhaps somewhat 'implementation defined', but that sucks roundly. 

As an example, this URL:

http://dotabuff.com/search?utf8=✓&q=PPD

This has a Unicode checkmark after utf8=.

How do we encode this in a zonefile? The checkmark is Unicode 0x2713, but
encoded as UTF-8 it is 0xe2 0x9c 0x93, or 226 156 147.

Should this therefore end up in a zonefile as
"http://dotabuff.com/search?utf8=\226\156\147&q=PPD" (or the PowerDNS
database, or the Lua plugin content field etc).

This would make the most sense to me, but it would be great to have
consensus.

At least if the RFC does not specify it, we should pick something. 

Thoughts?

PowerDNS
has an RFC7553 implementation in
https://github.com/ahupowerdns/pdns/tree/rfc7553 by te way.

	Bert

---

From: "Patrik Fältström" <[email-masked]>
Date: Wed, 17 Jun 2015 20:41:06 -0300

> At least if the RFC does not specify it, we should pick something.

The over arching issue here is that there is no right answer regarding non ascii in URIs.

A URI is a sequence of characters, but in HTTP the path must be ascii only, and can have %-encoded bytes -- without telling what charset is in use.

Because of this, I would say one can in RDATA store a URI in any charset, but when the URI is used, it should have 8-bit bytes %-encoded.

That said, libcurl do happily(?) send whatever bytes you give it over the HTTP connection, so %-encoding must happen before libcurl is used.

Earlier versions of the draft suggested one should use UTF-8 for the URIs in RDATA, which imply among other things the path should be %-encoded, but the question then was whether it also should be normalized etc etc...

And what if someone want non-UTF-8 in a URI?

What we have to remember is that whoever knows the URI and creates the RR also run the web site (for example) that is to be accessed. Because of this, the only question has to do with the %-encoding.

My PERSONAL preference would be to allow either %-encoded or non-%-encoded bytes in RDATA and that whoever consumes RDATA have to decide whether %-encoding should be applied or not. I would apply it if I where a programmer. :-)

Because of this, Bert, I would store things in UTF-8, and hope for the best :-)

   Patrik

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 18 Jun 2015 10:31:08 +0900

> encoded as UTF-8 it is 0xe2 0x9c 0x93, or 226 156 147.

See my first response.

						Masataka Ohta

---

From: Tony Finch <[email-masked]>
Date: Thu, 18 Jun 2015 10:53:37 +0100

> The over arching issue here is that there is no right answer regarding

What about RFC 3987 ?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
South Fitzroy: Northeasterly 6 to gale 8. Moderate or rough. Occasional
drizzle later in north. Good, occasionally poor later in north.

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 18 Jun 2015 08:10:21 -0300

>> The over arching issue here is that there is no right answer regarding

Not everyone have implemented it, and the consensus when creating the URI RR was that the RR should not be limited to only IRIs.

I see an issue with IETF not stating clearly IRI's (i.e. UTF-8 and then %-encoding) are in use is a problem both here and there. And such a statement is what I think is needed for not only the URI RR but also other cases.

   paf