Subject: [urn] (urnbis-15 review) the new abnormal: q-component and r-component

From: Sean Leonard <[email-masked]>
Date: Sun, 21 Feb 2016 09:47:27 -0800

The most significant technical changes in urnbis-15 are the treatment of 
q-component and r-component. As such, all of these changes are somewhat 
related.

I applaud the attempt to identify q-component with "?=" syntax and 
r-component with "?+" syntax. You get an E for effort.

Unfortunately, it is unworkable, and if there is one thing that I am 
going to be a stick-in-the-mud about, it's going to be this.

This WG concluded that q-component can contain the entire contents of a 
query component of a URI [RFC3986]. Therefore, q-component can contain 
any <query> production, including "?=" and "?+". It is entirely possible 
that some URI out there permits "?+" (and in fact, I am pretty sure that 
HTTP URIs permit such a sequence). This means that the proposed URN 
syntax is ambiguous.

Since q-component can be any query character and since q and 
r-components are in the query position of URIs, the *only* way to 
distinguish q-component unambiguously is to put q-component in the 
second position, and r-component in the first position. The URN syntax 
can then constrain r-component not to include "?", so "?" delimits 
r-component from q-component. This is entirely implementable.

       namestring    = assigned-name
                       [ "?" r-component [ "?" q-component ] ]
                       [ "#" f-component ]
       q-component   = query
       r-component   = resolverID ":" *( pchar / "/" )

That's the much-simplified ABNF. That means that if you want q-component 
and no r-component, q-component is preceded by "??". Okie-dokie.

If you really, really want "??" to delimit r-component, then you can 
have a tripartite construction:
??r-component
?q-component
??r-component?q-component

I think this is more convoluted than the proposed syntax and ABNF above. 
But it can be done.

***
It is said in Section 2.3 and q-component and f-component are 
namespace-independent. However, r-component is apparently *sometimes* 
namespace-independent, and sometimes namespace-dependent. Schizophrenia! 
Is it dependent, or independent??

If it is "sometimes dependent" (depending on the namespace definition), 
then...obviously it's namespace-dependent, because it always depends on 
whether the namespace says it's dependent or independent!

I believe that the WG really should resolve this before concluding this 
work...although it's not quite as much of a show-stopper as the ?+ ?= 
syntax issue above. Right now, r-components are namespace-dependent.

The question is whether we want to define things like resolverID to be 
"read" relative to the NID, or not. We have already concluded that a 
resolution service will take the NID, NSS, and r-component in when doing 
its job (but neither q-component nor f-component). From that standpoint, 
r-component will always be interpreted with NID in view.

Should a hypothetical resolverID "info" be defined essentially on a 
per-namespace basis, or should it be defined as a global resolution 
operation, "get some information about this URN", which may or may not 
apply to different URNs in different cases?

My view is that the people writing durable, universal naming schemes 
should not be required to be in the business (directly) of dictating 
what resolution services get developed or not, and vice-versa. I.e., the 
URN namespace registration template should not require new URN 
namespaces to specify a set of resolution ops.

 From that view, it follows that r-component is namespace-independent, 
in the sense that the maintainers of urn:isbn, urn:oid, urn:ietf, 
urn:nbn, etc. do not each get a stab at their own spins on the 
resolverID "foo" or "info" or "abcdefg". The namespaces of "resolverID" 
and therefore r-components more generally are global and orthogonal to 
the NID / NSS part.

We may wish to subdivide that global namespace into parts that 
incorporate NIDs, such as "*foo" being syntactic sugar for 
"urn-nid-specific-op-isbn!foo". (E.g., every URN namespace maintainer 
also gets a chunk of the resolverID namespace for its personal use for 
"free"--but this namespace is orthogonal to the URN NID/NSS namespace 
and does not have the same durability and uniqueness requirements.) I 
accept that such further definition is out-of-scope for this draft if we 
want to get it out in 2016. But we need to lay the groundwork.

***

I am very relieved to see that resolverID made it into this spec.

That being said, resolverID and the other part after the colon both 
under-deliver and over-deliver in material respects.

"?" is out as a character because that needs to be used to distinguish 
r-component from q-component.

I accept that resolverID is ASCII-only (no percent-encoding, and 
therefore no UTF-8). This makes things simpler. I also accept that 
trying to constrain resolverID to particular things (namely a list of 
keywords) is going to drag this work on too long.

That being said, there is no reason at this juncture to limit resolverID 
to LDH (letter digit hyphen). Since ":" delimits resolverID from the 
rest, we only need to remove ":". Allowing other characters will provide 
more flexibility. Therefore, proposal:

       resolverID    = unreserved / sub-delims / "@" / "/"

Since we are defining resolverID, the momentum is sufficient to 
structure the syntax of the other part. Although I am not combing 
through the history of the mailing list, I believe a handful of voices 
called for name/value pairs, and nobody strenuously objected.

Here is a proposal:

       r-component   = resolverID [ ":" r-params ]

       ; overall repertoire is *(pchar / "/")
       ; but "/" is not used--also "@" and ":" are not used
       ; because they are potentially used elsewhere
       ; and it's just better to limit parsing

       r-params      = r-param *( ( ";" / "&" ) r-param )

       r-param       = rp-name [ "=" rp-values ]

       ; pct-encoded prohibited in rp-name. Worth discussing
       rp-name       = *( unreserved / "!" / "$" / "'" /
                          "(" / ")" / "*" / "+" )
       rp-values     = rp-value *( "," rp-value )
       ; mostly isomorphic with encodeURIComponent except $ and +
       ; but that is intentional
       rp-value      = *( unreserved / pct-encoded / "!" / "$" / "'" /
                          "(" / ")" / "*" / "+" )

Compared with draft-14-seantek, the main difference is that I 
constricted rp-name so that it does not permit percent-encoding. I 
believe this is in keeping with the proposed resolverID being 
ASCII-only. In contrast, rp-value should be UTF-8.

Note: The draft-15 ABNF for r-component requires r-component to have 
both resolverID and at least one character after the ":". I believe that 
is a mistake: it should be permissible to specify resolverID without 
anything afterwards.

Sean

---

From: John C Klensin <[email-masked]>
Date: Sun, 21 Feb 2016 15:24:26 -0500

Sean,

I disagree with most of your note.  Much of this response is
intended ed to clarify the area(s) of disagreement, not to argue
with you about the points.  At least for the present, I'll leave
that to others... and to the co-chairs to decide how far we want
to go about the last issue.

--On Sunday, February 21, 2016 9:47 AM -0800 Sean Leonard
<[email-masked]> wrote:

> The most significant technical changes in urnbis-15 are the
> treatment of q-component and r-component. As such, all of
> these changes are somewhat related.
> 
> I applaud the attempt to identify q-component with "?=" syntax
> and r-component with "?+" syntax. You get an E for effort.
> 
> Unfortunately, it is unworkable, and if there is one thing
> that I am going to be a stick-in-the-mud about, it's going to
> be this.

Alternatives are welcome, but I believe we have been down this
path and that there is (at least so far) no consensus for
blowing off 3986 syntax entirely and that the WG concluded that
embedding r-component and q-component in the syntax of a 3986
"query" was the way to go.  If you want to re-argue that
question, I need to hear from the WG Co-Chairs before proceeding.

If we stick with "embedding in query", then, as I read the rule
in 3896, the <query> part starts after the first "?" and extends
to the end of the URI or the first "#".  That means that
_anything_ (other than "#"), including "+", "=", and more "?"s
with or without associated characters is, from a 3986
standpoint, just part of that <query> and poses no issues for
3986.

Now, I think there _are_ two issues with this idea of embedding
q-component and r-component in <3986-query>.    One arises if
one says "you can't only use the 3986 syntax, you have to
interpret it the way http-URLs do".  Put differently, there is
no requirement that the hypothetical generic URI parser be able
to divide the <3986-query> into r-components or q-components
because it doesn't know anything about them.   I believe the WG
has decided, in the "syntax only" notion, to not entertain that
position further and that, if you want to reopen it, you need to
convince the WG (and certainly the Co-Chairs) that you have
something new.   

The other issue is what we actually mean by "embedding", and is
a more complex question.  So far, the WG has believed that
simply delimiting the two substrings is sufficient.  However, if
we need to specify that any appearances of "?", or "?=" or, most
particularly, "?+" in those strings must be %-encoded and then
that the URN-parser needs to be able to decode one level of
%-encoding before putting the contents of the q-component into a
URL as a <query>, it seems to me that is easily accomplished in
terms of the document and not a big deal for a
URN-parser/evaluator.  It would add a bit to the confusion we
already have with IRIs and where %-encoding is required and
optional, but not very much.    So, while I think we need to
straighten out whether that encoding requirement is needed, I
don't see the problem as anything near "unworkable".

> This WG concluded that q-component can contain the entire
> contents of a query component of a URI [RFC3986]. Therefore,
> q-component can contain any <query> production, including "?="
> and "?+". It is entirely possible that some URI out there
> permits "?+" (and in fact, I am pretty sure that HTTP URIs
> permit such a sequence). This means that the proposed URN
> syntax is ambiguous.

At most, only if "?+" is not required to be encoded.  See above.

> r-components are in the query position of URIs, the *only* way
> to distinguish q-component unambiguously is to put q-component
> in the second position, and r-component in the first position.
> The URN syntax can then constrain r-component not to include
> "?", so "?" delimits r-component from q-component. This is
> entirely implementable.

I think the WG has been down that path, has concluded that there
are use cases for "?" in r-components (even if not in your model
of r-components, see below).  If we need to go through it again,
we need to hear from the co-chairs, especially (IMO) about
whether you are raising a new issue or reiterating an old one.

>...
> namespace-independent. However, r-component is apparently
> *sometimes* namespace-independent, and sometimes
> namespace-dependent. Schizophrenia! Is it dependent, or
> independent??
> 
> If it is "sometimes dependent" (depending on the namespace
> definition), then...obviously it's namespace-dependent,
> because it always depends on whether the namespace says it's
> dependent or independent!

I think that is correct and that, if we are going to stick with
resolverID, the only thing that is namespace-independent is the
interpretation of a given registered resolverID (see below).  If
the WG agrees and it is not already obvious, suggestions about
how to clarify the text would be welcome.
 
> I believe that the WG really should resolve this before
> concluding this work...although it's not quite as much of a
> show-stopper as the ?+ ?= syntax issue above. Right now,
> r-components are namespace-dependent.

The document's intent was that the only thing about r-components
that is namespace-independent is the ability to parse one into a
resolverID and a rest-of-the-r-component.   That was put in as a
compromise with, and concession to, you.  If it no longer works
for you, it can be taken back out, making r-components entirely
namespace-dependent.

> The question is whether we want to define things like
> resolverID to be "read" relative to the NID, or not. We have
> already concluded that a resolution service will take the NID,
> NSS, and r-component in when doing its job (but neither
> q-component nor f-component). From that standpoint,

See above.

> Should a hypothetical resolverID "info" be defined essentially
> on a per-namespace basis, or should it be defined as a global
> resolution operation, "get some information about this URN",
> which may or may not apply to different URNs in different
> cases?

Again, my understanding was that resolverIDs were to be defined
independent of the namespace/NID that uses them.  If that was
not the intent, then the text almost certainly needs work -- but
most or all of the justification for specifying the ability to
parse the r-component to identify the resolverID also disappears.

> My view is that the people writing durable, universal naming
> schemes should not be required to be in the business
> (directly) of dictating what resolution services get developed
> or not, and vice-versa. I.e., the URN namespace registration
> template should not require new URN namespaces to specify a
> set of resolution ops.
> 
>  From that view, it follows that r-component is
> namespace-independent, in the sense that the maintainers of
> urn:isbn, urn:oid, urn:ietf, urn:nbn, etc. do not each get a
> stab at their own spins on the resolverID "foo" or "info" or
> "abcdefg". The namespaces of "resolverID" and therefore
> r-components more generally are global and orthogonal to the
> NID / NSS part.

I hope that is consistent with what the text says.  However, the
intent of the text was that maintainers of particular namespaces
_do_ get to specify what resolverIDs are used with them, just
not how a given resolverID that they intend to permit is
interpreted.  If your notion that any resolverID can be used
with any NID/ namespace, then it takes on a lot of the
properties that 3986 assigned to <fragment>.   I believe the WG
has decided to not go there, but need the WG co-chairs for this.

> "urn-nid-specific-op-isbn!foo". (E.g., every URN namespace
> maintainer also gets a chunk of the resolverID namespace for
> its personal use for "free"--but this namespace is orthogonal
> to the URN NID/NSS namespace and does not have the same
> durability and uniqueness requirements.) I accept that such
> further definition is out-of-scope for this draft if we want
> to get it out in 2016. But we need to lay the groundwork.

This is, IMO, going _really_ far down the path that I thought we
agreed we weren't going to try to traverse... as a necessary
measure for getting finished any time in the near future.  If
you disagree, please try to convince the Co-Chairs and (idenally
from my standpoint) do it off list.

> I am very relieved to see that resolverID made it into this

Some of your comments above (and the next one) suggest that its
introduction raises too many other issues that need to be
resolved and that it should be taken back out.

> That being said, resolverID and the other part after the colon
> both under-deliver and over-deliver in material respects.
> 
> "?" is out as a character because that needs to be used to
> distinguish r-component from q-component.
> 
> I accept that resolverID is ASCII-only (no percent-encoding,
> and therefore no UTF-8). This makes things simpler. I also
> accept that trying to constrain resolverID to particular
> things (namely a list of keywords) is going to drag this work
> on too long.
> 
> That being said, there is no reason at this juncture to limit
> resolverID to LDH (letter digit hyphen). Since ":" delimits
> resolverID from the rest, we only need to remove ":". Allowing
> other characters will provide more flexibility. Therefore,
> proposal:
>...

I'm not going to engage on this unless others in the WG show an
interest in a discussion, one that is active and focused enough
to not hold us up for months debating a few characters for which
there is no demonstrated need beyond "we could...".  In defense
of that position, I think we should remember that it is much
easier to add (allow) a previously-prohibited character if it
turns out to be necessary than it is to delete one that turns
out to be a problem.

>...
> Since we are defining resolverID, the momentum is sufficient
> to structure the syntax of the other part. Although I am not
> combing through the history of the mailing list, I believe a
> handful of voices called for name/value pairs, and nobody
> strenuously objected.

Again, I believe the WG explicitly decided to not do this,
partially because it is questionable whether there is momentum
to do _anything_ beyond finishing up on the current path (if
then) and partially because it is not clear that there is
general agreement with your model about what "resolvers" are
about.

I believe the Co-Chairs have already rules further proposals in
this area off-topic.

    john

---

From: Sean Leonard <[email-masked]>
Date: Sun, 21 Feb 2016 13:04:41 -0800

On 2/21/2016 12:24 PM, John C Klensin wrote:
> with you about the points.

Well let's clarify the disagreement then...

To keep things clear I am going to respond separately to the three major 
points...

On the namespace-independence of resolverID:

>   At least for the present, I'll leave
> interpreted.

I am confused with these two statements/positions and do not see how to 
separate them.

I think an example from you would be very helpful.

As you wrote: "maintainers of particular namespaces [get] to specify 
what resolverIDs are used with them". And: namespace maintainers do not 
get to specify "how a given resolverID that they intend to permit is 
interpreted".

So the maintainer of urn:isbn can say the resolverID "info" can be used, 
and "get" cannot be used. The maintainer of urn:oid can say the 
resolverID "info" can be used, and "get" can also be used.

But why do they say some resolverIDs can be used or not, when they have 
no way to promulgate what it means?

It is like saying: "You can use the number 6 in this protocol, but not 
the number 7. Oh, but you can do whatever you want when you receive a 6."

Usually the party that specifies the values, also specifies the meanings 
of the values.

I actually do not care whether the resolverID is namespace-dependent or 
namespace-independent. As I have pointed out, a resolver receives both 
the resolverID and the NID (plus the NSS, of course), so at the level of 
resolver behavior, the behavior will always be namespace-dependent 
because it depends on the identity of the thing being named.

Right now the specification equivocates because sometimes it says 
independent and sometimes it says dependent. That is a problem.

Upon reasoning it through, my position was:

"people writing durable, universal naming
set of resolution ops."

This is based on making it easier to register URNs. If resolverIDs 
(resolver operations) are namespace-dependent, we must expect the URN 
registration template to define resolution a lot better. That is more 
work. And it may change over time based on different parties' needs.

But if we want to give the URN namespace maintainers a lot more work to 
do, I am fine with that. Just pick one and be consistent throughout the 
document.

> If your notion that any resolverID can be used
> properties that 3986 assigned to <fragment>.

It's more like, "do the namespace maintainers specify the resolverIDs, 
or not". Does the urn:isbn maintainer maintain a list of resolverIDs? 
How about urn:ietf? Where is they?

(And again, it's fine if they do...but either the list needs to be in 
the registration template, or the process of management needs to be 
discussed in the registration template, so the list is stored elsewhere.)

In some sense this is a mirror image of Julian Reschke's discussion 
about "what is the base URI of a URN in the XML/HTML base usage". The 
answer is sort of, you can do that, but it doesn't make sense, so don't 
worry about it...(at least from the standpoint of this document).

Sean

---

From: Sean Leonard <[email-masked]>
Date: Sun, 21 Feb 2016 13:22:32 -0800

On 2/21/2016 12:24 PM, John C Klensin wrote:
> a discussion, one that is active and focused enough

As an implementer, that is not my position. It is one thing to say that 
a character is "RESERVED"--it's another thing entirely to prohibit it in 
the ABNF.

Once this thing gets published, people will write code that implements 
the ABNF. You can't add characters so easily.

The limitation of resolverID to LDH is arbitrary and (as far as I can 
tell, since the discussion has come up) capricious. You could say "we 
could limit it", but you could also easily say "we could expand it". Up 
until now the set of characters has not gotten much airplay.

If you want to stuff more in the resolverID production than LDH, the 
only way to do it based on the current ABNF in draft-15 is Punycode, 
just like DNS.

Keep it expansive. If you want you can add a note to the effect of, 
"resolverID is supposed to be limited to LDH, other characters are 
RESERVED or may have RESERVED purposes in other specifications." (I do 
not advocate for this position, but I am pointing out how to do it 
correctly, rather than put the limit in the ABNF. Note that is how RFC 
2141 dealt with the issue of characters such as "?" and "#".)

> Again, I believe the WG explicitly decided to not do this,

My recollection is that there were calls for the structure of name/value 
pairs that got a lot of serious consideration. If we can agree on 
name/value pairs then we will make serious technical progress.

The discussion faltered on a *registry* of r-component name/value pairs, 
but that was about reserving specific names, not about the syntactic 
structure.

Refs:
http://mailarchive.ietf.org/arch/msg/urn/cOq7rl_zd4pCPdZ-kpLm-ujBvoM
http://mailarchive.ietf.org/arch/msg/urn/78hKQSNHvlBx1RJnPUOczkykkFs

If you can point to a definitive decision not to do this, then I will 
desist on the point.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Sun, 21 Feb 2016 13:42:26 -0800

On 2/21/2016 12:24 PM, John C Klensin wrote:
> "query" was the way to go.

John, I want to make sure that we are discussing the exact same thing 
since this comment makes it seem that we have different objectives. We 
have the same objectives. The whole purpose is to put r-component and 
q-component into the same protocol slot as the query production of RFC 3986.

> standpoint, just part of that <query> and poses no issues for

Yes. 100% in agreement.

> because it doesn't know anything about them.

That is correct. Neither of us is saying anything about HTTP or what 
have you at this point. It's all in a <query> production of RFC 3986.

> URN-parser/evaluator.

Ok well here we have a substantive difference, that I hope we can tease out.

I am not sure that "So far, the WG has believed that simply delimiting 
the two substrings is sufficient." Some in the WG have said this 
(references, please?); my position is that delimiting is not sufficient. 
As you pointed out: if you delimit them with ?= ?+ etc., you must also 
%-encode the constituent parts. This means multiple levels of 
%-encoding, since % encoded characters will themselves get % encoded to %25.

History has shown that double-encoding things is a security disaster. 
There is a whole class of security vulnerabilities where implementations 
fail to properly double-encode or double-decode things. Furthermore, 
it's just not readable. People are going to transcribe these things 
by-hand too, especially for library things.

>    It would add a bit to the confusion we

If it is conceded that an additional layer of %-encoding is required by 
using ?= and ?+ delimiters (because, as I just said, a malicious or 
otherwise specially crafted q-component could contain ?+ thus switching 
the URN parser into r-component-parsing mode), then one also concedes 
the point that the current draft-15 text--which makes no mention of 
additional %-encoding--is "unworkable".

> of r-components, see below).

To be clear, ? in q-components is foreseeable...in that q-component = 
any URI query, so ? can appear in the wild. I.e., it will appear, 
especially in deliberately crafted payloads.

We have control however over r-component, by prohibiting ? or mandating 
that ? be %-encoded as %3F. (These are tantamount to the same thing.) 
There are lots of other delim characters that can be imbued with 
reserved purposes, within an r-component production.

Sean

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 22 Feb 2016 14:21:42 +0900

> I think an example from you would be very helpful.

I think I understand what John is trying to say: If there's a resolverID 
"info", and that's (for simplicity's sake) defined to return info about 
the author(s) and the length of the document in bytes, then it's not 
possible for a namespace maintainer to say that for the particular 
namespace, the "info" resolver also returns the date of creation, or 
doesn't return the length in bytes, and so on. In other words, opt-in 
for each resolverID is binary.

That still leaves us with the problem below.

> work. And it may change over time based on different parties' needs.

I guess I'm with Sean here. Sean maybe implied this, but I didn't see it 
stated explicitly: It would make introduction of new resolution services 
(which might perfectly well be able to deal with preexisting URN 
namespaces) a lot more difficult. We don't have any magic to update 
specs or registrations automatically.

So at the most, I'd want to limit us to something like "Namespace 
registrations MAY indicate some of the types of resolvers suitable for 
the namespace (by listing the relevant resolverIDs). Such a list does 
not restrict the use of resolvers to only the listed types."

Regards,    Martin.