Subject: Re: [urn] Think about r-components and q-components differently

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 26 Oct 2015 19:58:09 -0600

On 9/23/15 5:22 PM, Sean Leonard wrote:
> URNies:
>
> The introduction of r-components and separating out directives to the
> resolver, from directives to the resource*, is a welcome development.
>
> For my part I want to put a big +1 to putting the r-component before the
> q-component. I.e.:
>
> urn:isbn:0385537670??service=showscanned?dpi=600#chapter2

I have gone back and forth on this. Conceptually I agree with you that 
it seems cleaner to have the r-component first.

I think we need to have the chairs tell us how they want to proceed here 
(see also John Klensin's message from August 13, 2015).

Peter

---

From: Melinda Shore <[email-masked]>
Date: Mon, 26 Oct 2015 18:07:09 -0800

> (see also John Klensin's message from August 13, 2015).

I am generally extremely loathe to have chairs resolving
technical questions in the absence of an emergency.  That
said, I think we have an emergency.  I think we need to hear
from John if he'd block going ahead with r-then-q.

Melinda

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 26 Oct 2015 20:10:29 -0600

> from John if he'd block going ahead with r-then-q.

Sorry, I was not clear: it strikes me that we might need a consensus call.

Peter

---

From: John C Klensin <[email-masked]>
Date: Tue, 27 Oct 2015 15:24:06 -0400

--On Monday, October 26, 2015 18:07 -0800 Melinda Shore
<[email-masked]> wrote:

>> I think we need to have the chairs tell us how they want to
>> proceed here (see also John Klensin's message from August 13,
>> 2015).
> 

Melinda,

I hope to catch up in the recent flurry of messages within the
next day or three but, unfortunately, tried to reserve time to
review a complete -14 posted some time ago and then immediately
before the deadline and therefore have not had time to parse
these messages.  However, with the understanding that there
could be something in the message threads that would cause a
revelation and result in a change of mind...

I continue to think that r-then-q is a bad idea. I won't repeat
the reasons, which I have explained at length.   I also believe
that any argument that hinges on whether something is more
likely to be done by left to right or top-down parsing or
regular expression matching is probably a sign that we've got
the basic design idea wrong and should be looking for other
alternatives.

However, I agree with others that the cases in which the choice
is likely to lead to a big issue are going to be very rare and
probably represent bad judgment by whomever is designing the
namespace or whatever the URN points to.  I therefore do not
consider one resolution or the other of this issue to be a
showstopper or "known defect" and think it is more important to
get this work finished than to spend significantly more time and
energy trying to find strong consensus on an optimal solutions.
So I'm not going to try to block either choice.

That said, I do have one requirement and one suggestion:

(1) Whichever ordering is chosen, the ABNF must be correct,
unambiguous, or conform to RFC 5234.   As long as they are
unambiguous, a closed set of productions would be fine, a more
open set of productions plus prose restrictions would be fine,
but some "pick the production that most nearly resembles the
string" handwaving (akin to part of 3986) is not.

(2) I won't try to insist but I think it would be wise to
include a strong warning in the text about embedded "?"
characters inside the r-component and q-component strings and
about decoding implications of embedded %3F if the intention is
to embed a substring of the URN to some other processor (e.g.,
in a URL) and whether such a escape should be decoded before
embedding.   If r-component comes first, that also implies a
warning about the use of embedded "??".  The syntax must be
clear enough that any such construction has one and only one
interpretation, but it would be reasonable to also suggest that
such constructions should be avoided if possible because sloppy
implementations seem inevitable and almost certain to trap the
unwary and those whose constructions depend on exact and
accurate interpretation of this spec and the relevant rules from
3986.  I'd be happy to try to construct text once I see -14.

    john

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 28 Oct 2015 14:38:57 -0600

On 10/27/15 1:24 PM, John C Klensin wrote:
> I hope to catch up in the recent flurry of messages

Sorry about that - I seem to be processing IETF work in batch mode.

> within the
> the reasons, which I have explained at length.

For other folks here, the primary message seems to be:

http://www.ietf.org/mail-archive/web/urn/current/msg03025.html

And, secondarily:

http://www.ietf.org/mail-archive/web/urn/current/msg03120.html

>  I also believe

Thsoe are good points that we need to take into account before we're done.

Peter

---

From: John C Klensin <[email-masked]>
Date: Wed, 28 Oct 2015 23:39:37 -0400

--On Wednesday, October 28, 2015 14:38 -0600 Peter Saint-Andre
<[email-masked]> wrote:

>...

>> the basic design idea wrong and should be looking for other

FWIW and with apologies for this being so late, I've been
(again, since the prior note) trying to think through the
reasons we are continuing to have an "r-component or q-component
first" debate, one that has been going on for months with no
obvious reason for strongly favoring one option over the other
and doing so in the light of the "basic design idea wrong"
hypothesis above.

So, stepping back some distance...

The problem we are having ultimately stems from trying to use
"?" (derived from the 3986 <URI> production as an introducer to
<query>) for the q-component and "??", a two-character URI
invention, as an introducer for r-component, rather than
treating both q-component and r-component as subsidiary to what
3986 calls "query".  That doubled question mark causes many
problems with either ordering although the problems differ.  In
particular, it makes the ABNF hard to write and it creates edge
cases that are likely to confuse implementations and/or users.

If the problem really lies with that doubled question mark, then
suppose we do something different.  In particular, we find two
special characters that are now allowed within <query> (that is
just about anything other than "#").  We could then define the
relevant syntax as follows (definitions for <URI>, <query> and
<pchar> copied from the designated sections of 3986 for
convenience):

	URI    = scheme ":" hier-part [ "?" query ] [ "#"
	  fragment ]
	    ; section 3 intro
	query  = *( pchar / "/" / "?" ) 
	    ; section 3.4   
	pchar  = unreserved / pct-encoded / sub-delims / ":" /
	  "@"
	    ; section 3.3

 Now, since the handling of <query> in 3986 is opaque as long as
the syntax above is satisfied, we define <namestring) (from
draft-ietf-urnbis-rfc2141bis-urn-13, section 3) as (I think
there are shorter ways to do this, but the below should be at
least absolutely clear for discussion purposes)

	namestring  = assigned-name
	             [ query-style-string ]
	             [ "#" f-component ]  
	query-style-string = ( q-string [ r-string ] )
    / ( r-string [ q-string ] )
	q-string = "?" specialChar1 q-component
	r-string = "?" specialChar2 r-component

Now, unlike some of what we've tried to do, it is a lot less
complicated than it looks.  q-component and r-component can be
defined as they are now in 2141bis.  There is one restriction
that, in simple form, is that neither can contain "?" followed
by <specialChar1> or <specialChar2>.  Or one could get fancy and
prohibit "?" and <specialChar2> inside the q-component and "?"
and <specialChar1> inside the r-component.  I don't personally
recommend fancy -- if the special characters are chosen
reasonably, the number of cases where their use following "?"
within a string are going to be sufficiently few (and bizarre)
that forcing escaping is reasonable and probably even a good
idea (more on this below).

The above permits q-component or r-component without the other
and without creating edge cases.  It also permits them to appear
in either order if both appear.   It (deliberately) has one
interesting and useful side-effect: the first "?" in a URN must
be followed immediately by either specialChar1 or specialChar2
or it is invalid (independent of namespace).  That gets rid of a
number of edge cases that have plagued the "?" and "??" model.

specialChar1 and specialChar2 should be chosen to minimize the
likelihood of collisions with URL queries.  I'd guess that most
of the characters other than left parentheses, left brackets,
and single-quote listed for <sub-delims> in 3986 would be
reasonable candidates.  To the extent to which queries
ordinarily take the form 
   ? [SP] phrase ...
or
   ? [SP} keyword = value ...
forms like 
   ?<selected-sub-delim>
are just not going to be an issue.

Now, this does mean that 2141bis is going to need to be explicit
about how a URN processor whose target is a URL extracts and
converts a q-component into the query part of that URL.   That
rule is essentially that the q-component _is_ the query, but the
"?" has to be re-supplied (a requirement if we are going to use
the current 2141bis syntax anyway because the question mark is
not part of the q-component itself.   So nothing really changes.
I think that 2141bis will need a paragraph or two to that effect
anyway, just to get the level of clarity I think we need.  That
paragraph would need to specify that, if "?" inside a
q-component is followed by the %-encoded form of specialCharX,
it needs to decode it in the process of building the query.
I'd prefer to do that by requiring that "?" be encoded for that
case too, but I don't feel strongly about it.  Again, some
version of that language would still need to be present for ?%3F
and %3F%3F inside q-components for the syntax now used in
2141bis, so this is not an extra problem.

In practice, this would give us something that would look like
(with one largely arbitrary choice of spacialChar1 and
specialChar2):

  URN:NID:NSS?)q-component?]r-component#f-component

Again, with q-component and r-component in whatever order amuses
the user until the WG decides to make that choice, which, in
this approach, is arbitrary and aesthetic, not technical.   For
the hypothetical generic URI processor, that string decomposes
into
   scheme = "URN"
   hier-part = "NID:NSS"
   query = ")q-component?]r-component"
   fragment = "f-component"

all of which is perfectly valid and straightforward.

     john

---

From: Sean Leonard <[email-masked]>
Date: Sat, 31 Oct 2015 15:15:36 +0900

Hello:

On Oct 29, 2015, at 5:38 AM, Peter Saint-Andre <[email-masked]> wrote:

>>> […] I think we have an emergency. […] block going ahead with r-then-q.
>> 
>> Melinda,
>> 
>> […]
>> warning about the use of embedded "??”.  The syntax must be
>> interpretation, […]

A very simple proposal:

Remove the q-component.

Seriously. At this point, URNBIS draft punts on what the q-component means…it basically just says that the q-component should be passed along and tacked on to the resolved URLs (URI) as-is.

How is that going to work if the resolver returns data: URLs (yes, they are “URLs”, see RFC 2397)? How about if the resolver returns mid: URIs? What about ldap: URIs? What about magnet: URIs? a mixture of all of those? The syntax and semantics of q-components in these different URIs are markedly different.

The r-component makes a lot of sense. Have the resolver translate the URN to other URIs (URLs, whatever). Note that the resolver can always return a data: URL (URI) so this whole worry about returning resources or resources to resources or pointers to resources to resources or what have you seems to be moot. Trying to “force” the resolver to interpret the semantics of the r-component, but pass the q-component along, does not strike me as workable.

Thanks,

Sean

PS LDAP URIs can contain ?? or more in succession--that has a specific meaning. Something to think about. [RFC4516]

---

From: John C Klensin <[email-masked]>
Date: Sat, 31 Oct 2015 12:01:25 -0400

Sean,

I think we are at risk of going around in circles... again.
I'll leave it in the co-chair's hands what, if anything, to do
about that, but suggest that, each time it happens, various
people get demoralized and, even if they later come back, work
grinds to a halt for an extended period.

Specifics below.

> Seriously. At this point, URNBIS draft punts on what the
> q-component means…it basically just says that the
> q-component should be passed along and tacked on to the
> resolved URLs (URI) as-is.

I don't think "as-is" completely and unambiguously works (see
prior note), but it feels to me as if we have had endless
discussions about why something like a q-component is important
and necessary for some significant applications (some of which
do not involve mapping from URN to URL).  If 2141bis is not
clear enough about that, suggest text or at least what should be
fixed and where (see below)

> How is that going to work if the resolver returns data: URLs
> (yes, they are "URLs", see RFC 2397)? How about if the
> resolver returns mid: URIs? What about ldap: URIs? What about
> magnet: URIs? a mixture of all of those? The syntax and
> semantics of q-components in these different URIs are markedly
> different.

We have got all kinds of situations in which "resolving" a URN
means different things (or nothing or nonsense) depending on the
namespace.  Mapping to some other flavor of URI is just one of
the cases [1].  And certainly we've reached the conclusion
several times that, no matter what the original intent (with or
without a well-developed URC infrastructure), it is not, today,
a requirement of the "urn:" scheme that all such URNs resolve to
URLs (or even URCs).  The generic answer to your questions about
is "If anyone defines a namespace whose URN resolves to one of
those schemes, it will get nonsense. An implementation of those
schemes had better be as ready to defend against that as it
would defend against any other nonsense".

> The r-component makes a lot of sense. Have the resolver
> translate the URN to other URIs (URLs, whatever). Note that
> the resolver can always return a data: URL (URI) so this whole
> worry about returning resources or resources to resources or
> pointers to resources to resources or what have you seems to
> be moot. Trying to "force" the resolver to interpret the
> semantics of the r-component, but pass the q-component along,
> does not strike me as workable.

That is more or less where we started.  My impression is that
the WG concluded, rather strongly, that it was important to
distinguish between instructions/requests to the resolver and
instructions/requests about the object if there was one and to
do so with syntax that applied to all URNs (although whether the
syntax was useful/usable or not would be per-namespace).  Like
you, I'm not sure that, in some hypothetical general case, the
distinction is always that clear.  But, fwiw, I'm not convinced
that going over that question again would be of any benefit.

> PS LDAP URIs can contain ?? or more in succession--that has a
> specific meaning. Something to think about. [RFC4516]

My guess is that, if we look at the whole space of URI schemes
that are documented and/or in use, we will find almost every
syntax trick imaginable.  I do think that a known specialized
use of "??" strengthens the case for ?<special-delimiter> where
both q- and r-components have those delimiters and neither is
"?" or for nesting q- and r-component within what 3986 thinks of
a <query>.  We also need to do a further clarity check on
2141bis in this area and probably change "as-in" to an explicit
mapping, especially given the potential for interactions of IRIs
[2].

>From my perspective, the problem goes back to 3986 and, in some
cases, its predecessors: a lot of characters are given special
interpretations, others are given special interpretations only
in specific syntactic contexts, and still other are or are not
delimiters based on the scheme.  In addition, a lot of the
language and assumptions are based on URLs that map to locations
or objects but not to other, non-embedded URIs.  As I have said
before, I now believe, in retrospect, that it was a mistake to
create a "URI"-thing rather than creating URLs, generic (not
just one scheme) URNs, and maybe URCs that are distinguishable
by basic syntax, with separate grammars for each.  But that
horse left the barn before RFC 1737 was published and, if there
was any possibility of retrieving it, that window closed years
ago.  

I think we are at a fork in the virtual road.   On one path, we
just figure out how to make separate q- and r-components work,
making sure that 2141bis contains appropriate qualifying
language.  On the other, we take a more extreme version of
Keith's "URNA" proposal, depreciate URNs as having gotten just
much too confused and now representing a bad combination of
incompatible concepts, and then start efforts to define a new
set of typologies for generic name-type things that may or may
not define or point to resources.   Or we can stand in the road,
looking at the fork, and either jumping from one foot to the
other or running in small circles until some truck comes along
and puts an end to the discussion.   

As a specific example of a truck-instance, suppose the ART ADs,
sometime between now and IETF 95, come to the WG Co-chairs and
say "given the WG's record of setting schedules and missing them
and the observation that you are having some of the same debates
now that you were having three or four years ago (remember the
WG was created in the last quarter of 2010), is there any
plausible reason to expect that it will converge and produce
consensus results any time soon, because, if not, we are going
to shut you down".   If you were one of the co-chairs, how would
you answer?

best,
     john

[1] Given language in 2396 and 3986 that essentially says that
some of what had been called URLs are now URIs, it is hard for
me to make strong inferences about what something is or is not
from the use of the term "URL" in various contexts.

[2] IRIs may be another example of the overreaching to has
gotten us into trouble... it is likely that they should be
called "IRLs" and that the internationalized analogy to URNs
("IRN"s?), if there is one, should be subject to different
rules, especially for the cases in which they are expected to
map into URLs.