Subject: Re: [urn] encoding, component order, and related issues

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 1 Sep 2015 20:48:28 -0600

On 8/14/15 6:23 PM, Keith Moore wrote:
> I never was convinced of a consensus on this issue (though it's hard
> to tell when there are so few active participants and people
> sometimes take a long time to reply).  But I'm also no longer
> convinced that this is a huge problem.

Keith, would you mind explaining exactly what 'this' refers to above? Do 
you think the lack of consensus is not a huge problem, that the number 
of active participants is not a huge problem, or that ordering the 
components as q-then-r (instead of r-then-q) is not a huge problem?

Peter

>
> Sent from my iPhone
>
>> On Aug 14, 2015, at 12:46 AM, John C Klensin <[email-masked]>
>> wrote:
>>
>>
>>
>> --On Thursday, August 13, 2015 20:52 -0600 Peter Saint-Andre - &yet
>> <[email-masked]> wrote:
>>
>>>> But I also realize that this is at least the third time I've
>>>> tried to fix this, and that the glitches that this causes will
>>>> probably be infrequent.   I also strongly suspect that this
>>>> will come up again in IETF-wide Last Call because I expect the
>>>> web types will want to go over this with a fine-toothed comb.
>>>> So while I still don't understand why you so strongly insist
>>>> that r-component has to follow q-component,
>>>
>>> My sense is that John feels this was a closed issue and that the
>>> WG had reached agreement on q-then-r, so why revisit it?
>>
>> Yes.  I'm happy to reconstruct and explain again the reasons for
>> preferring q-then-r, including why I think the "copying" argument
>> is mostly bogus in practice,  but I'm disinclined to spend
>> everyone's time doing so unless either the co-chairs say "actually
>> there was no conclusion earlier" or someone other than Keith feels
>> strongly about the other order and says so.
>>
>> john
>>
>>
>>

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 1 Sep 2015 20:45:51 -0600

> convinced that this is a huge problem.

Keith, what exactly does 'this' refer to above? Do you think the lack of 
consensus is not a huge problem, that the number of active participants 
is not a huge problem, or that ordering the components as q-then-r 
(instead of r-then-q) is not a huge problem?

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Keith Moore <[email-masked]>
Date: Tue, 1 Sep 2015 22:54:31 -0400

On 09/01/2015 10:45 PM, Peter Saint-Andre - &yet wrote:
> Keith, what exactly does 'this' refer to above? Do you think the lack 
> of consensus is not a huge problem, that the number of active 
> participants is not a huge problem, or that ordering the components as 
> q-then-r (instead of r-then-q) is not a huge problem?

I am no longer convinced that the inability of a q-component to 
represent '??' without percent-encoding one or both of those characters 
is a huge problem.

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 1 Sep 2015 21:20:33 -0600

Hi Keith,

Thanks very much for the comprehensive review, and especially for the 
suggested text. Because your email is so long, I am going to break my 
reply into smaller chunks.

On 7/1/15 4:03 PM, Keith Moore wrote:

> *1.*  Section 3, first paragraph:
>
> The rationale for the syntax changes made in this document needs some work.
>
> IMO, "Semantic flexibility in the use of URNs within particular
> applications" is not a desirable goal for URNs, or maybe I just
> interpret those words differently than intended.  We need for URNs to be
> interpreted uniformly from one (client) application to the next.
> Perhaps this is just an ambiguity that needs to be cleared up, but the
> need to have client implementations interpret URN components uniformly
> from one client to the next (i.e. interoperability) should inherently
> constrain the semantics of URN components to some degree.

Agreed. The point, as I understand the quoted text, is that members of 
the URN-using community felt the need for greater flexibility in URN 
syntax (compared to RFC 2141) so that URNs could meet the needs of 
existing and future applications - more concretely, this implied the 
ability to use q-components, f-components, and now r-components. So this 
is about 2141 syntax vs. 2141bis syntax, not about "anything goes" 
within URNs.

> (I'll have separate comments for urnbis-semantics.)
>
> CURRENT:
>
> 3.  URN Syntax
>
>     The syntax of URNs as provided in [RFC2141] was defined before the
>     updated specification of URIs in [RFC3986].  In order to ensure
>     consistency with the URI syntax as well as semantic flexibility in
>     the use of URNs within particular applications (see
>     [I-D.ietf-urnbis-semantics-clarif] for further discussion), this
>     specification does the following:
>
>
> RECOMMENDED:
>
> 3.  URN Syntax
>
>     The syntax of URNs as provided in [RFC2141] was defined before the
>     updated specification of URIs in [RFC3986].*In order to facilitate****the use of URNs with parameters similar to URI queries and fragments,****permit parameters influencing URN resolution to be included with****URNs, and ease use of URNs with identifier spaces that **include****the '/' character,*  this specification does the following:

Your proposed text is helpfully concrete, so +1.

> (see also my note below about including "~" and "&" in NSSs)
>
> *2.*  Problems with grammar in section 3
>
> a) The grammar is both ambiguous (in that it permits the same string to
> be parsed in multiple/conflicting ways) and inconsistent with the
> text.   In particular, if "??" separates a q-string from r-string, then
> q-string should not permit "?" to appear twice in succession, and the
> currently specified grammar does.
>
> b) The grammar refers to "q-string" etc. but the text refers to
> "q-component" etc.  I could find no reason for the discrepancy.

Editorial oversight, I think.

> c) Also, the encoding problem for q-component still remains.    See below.
>
> CURRENT:
>
>
>        namestring    = assigned-name
>                        [ "?" q-string ]
>                        [ "??" r-string ]
>                        [ "#" f-string ]
>        assigned-name = "urn" ":" NID ":" NSS
>        NID           = (alphanum) 0*30(ldh) (alphanum)
>        ldh           = alphanum / "-"
>        NSS           = pchar *(pchar / "/")
>        q-string      = pchar *( pchar / "/" / "?" )
>        r-string      = pchar *( pchar / "/" / "?" )
>        f-string      = fragment
>
>     Note: The character "?" can be used without percent-encoding inside
>     q-components, r-components, and f-components.
>
>
> ALTERNATIVE 1 (keep order and separators the same but fix ambiguity):
>
>
>        namestring    = assigned-name
>                        [ "?" q-*component*  ]
>                        [ "??" r-*component*  ]
>                        [ "#" f-*component*  ]
>        assigned-name = "urn" ":" NID ":" NSS
>        NID           = (alphanum) 0*30(ldh) (alphanum)
>        ldh           = alphanum / "-"
>        NSS           = pchar *(pchar / "/")
>        q-*component*    =*pchar *( pchar / "/" / ( "?" ( pchar / "/")))*
>        r-*component*    = pchar *( pchar / "/" / "?" )
>        f-*component*    = fragment
>
>     Note: The character "?" can be used without percent-encoding inside
>     q-components, r-components, and f-components, but two adjacent "?"
>     characters MUST NOT appear inside a q-component.
>
>
> Basically, the change specifies that any "?" in a q-component must be
> followed by either a pchar or a "/".
>
> However, this suggested grammar still has the encoding problem that I
> mentioned earlier:   If q-component is interpreted as a query to be
> attached to a URL (which is what I thought we had agreed to, at least
> for URNs that resolve to URLs), there need to be clear rules for copying
> or converting that q-component text to the URL query text.   The above
> grammar does not permit multiple "?"s to appear adjacent in a
> q-component.   This implies that if there's ever some need to bundle a
> URN with a query that contains "??", the "??" must somehow be encoded so
> that (a) it doesn't violate the URN syntax grammar, and (b) the original
> query string is recovered when bundled with the URL.
>
> Simply assuming percent-encoding doesn't fully specify a solution,
> because while a query that contains "??" could be encoded as %3F%3F, a
> resource might reasonably interpret %3F%3F differently from "??"
>
> This is one reason that I believe it's much simpler, and more likely to
> be implemented correctly, to not constrain the syntax of q-components
> beyond the syntax of RFC 3986 queries.   And that implies either (a)
> putting r-component somewhere else than following q-component [e.g.
> ALTERNATIVE 2 below], or (b) using a different character sequence to
> separate r-component from q-component [e.g. ALTERNATIVE 3].   Then we
> can specify that if a URN with a q-component resolves to a URL, the
> q-component is simply copied to the URL query.
>
> RECOMMENDED ALTERNATIVE 2 (switch order of r-component and q-component):
>
>        namestring    = assigned-name
>                        *[ "??" r-component ] *                       *[ "?" q-component ] *                       [ "#" f-*component*  ]
>        assigned-name = "urn" ":" NID ":" NSS
>        NID           = (alphanum) 0*30(ldh) (alphanum)
>        ldh           = alphanum / "-"
>        NSS           = pchar *(pchar / "/")
>        q-*component*    *= *( pchar / "/" / "?" )*
>        r-*component*    *= pchar *( pchar / "/" )*
>        f-*component*    = fragment
>
>     Note: The character "?" can be used without percent-encoding inside
>     q-components, r-components, and f-components.

I have come to think that placing the r-component first has several 
advantages - it keeps the new URN-specific component syntactically 
separate from the existing URI components, everything after the 
r-component can be passed directly to a standard URL or query-string 
parser (without splitting things up or pulling out this strange new 
r-component thing from the middle), and as you note it enables the first 
character of a q-component to be "?".

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 1 Sep 2015 21:21:34 -0600

> represent '??' without percent-encoding one or both of those characters

OK, thanks for the clarification.

Peter

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 3 Sep 2015 17:53:34 -0600

Here is the second installment of my reply...

On 7/1/15 4:03 PM, Keith Moore wrote:

> *3.*  Section 3.2, 2nd paragraph:
>
> This document modifies the syntax of the NSS to allow the following
> characters: "/", "~", and "&".
>
>
> Are there reasons to permit "~" and "&", and have we sufficiently
> investigated the implications of doing so?

I think the idea was: is there a reason to *not* permit "~" and "&"? 
This is an artifact of trying to align with the URI spec as much as 
possible, in this case by reusing the 'pchar' rule.

>  I am wondering whether
> this will break parsers for no good reason.  I have seen URLs which
> "~" was encoded to "%7E" (even though this shouldn't be necessary)
> and "&" to "%26" (which also shouldn't be necessary except in a query
> that uses HTML FORM encoding for HTTP GET).

It's not our job to design around broken parsers. That way lies madness.

> *4. * This document needs to clarify (somewhere) that the change to
> the URN grammar DOES NOT change encoding rules for existing URN
> namespaces, if there are existing namespaces that are used outside of
> the URN context, and in which use any of "/", "~", or "&" are allowed
> to appear in those names in native form.

Although I have no objections to including such text, the universe of 
URN namespaces is small enough that we could determine if any such 
namespaces exist:

http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml

> *5.*  Also, something that I think could possibly use a bit more
> explicit attention in the registration section and perhaps elsewhere:
> In some sense there are two kinds of URN namespaces: those that exist
>  solely as URNs, and identifier spaces that exist independently of
> URNs.   The latter kind of identifier has a "native" form and a URN
> form, and its URN namespace registration should specify an algorithm
> for converting from native form to URN form.   Ideally this algorithm
>  produces a distinguished form of its input when converting it to URN
>  format - this makes it easier to build generic services that answer
>  queries for arbitrary URNs.   If the algorithm is just to use
> percent-encoding, that's fine, but IMO this should be explicitly
> specified in the namespace registration. I don't think we should
> unilaterally impose this algorithm for all identifier spaces.

But do you think we should recommend it? This feels like one of those 
situations where the creativity of the namespace registrant could lead 
to bad results.

> Client software that just handles URNs doesn't need to worry about
> this.   Such software should always see the URN form of such
> identifiers and should be able to handle all URNs in a uniform
> manner.   However, software that allows such identifiers to be input
> directly (e.g. without the urn:xxx: prefix) and converts those
> identifiers to URNs, needs to be aware of the conversion rules for
> each URN namespace that it accepts.

Aren't we verging close to violation of the principle of uniformity 
here? I'd at least prefer to say that you SHOULD use percent-encoding 
(unless you have good reason not to, and please document your reasons, 
such as having an existing encoding rule as you describe below).

> Anyway, I think it would help to have some background text early in
> the document that explains this, and one or two specific items in the
>  registration template

That does sound helpful.

> (I can try to come up with text if desired, but I didn't want to
> delay this review any longer.)
>
> (Also, much of this item is also reflected in the next item.  Sorry
> for not merging them; I wanted to get this out the door.)
>
> *6.*  Section 3.2, 3rd paragraph:
>
> I'm concerned that telling implementors to use %-encoding of
> disallowed characters that appear in NSS might conflict with an
> existing URN namespace's encoding rule.  Also, %-encoding is a mess,

True. But it's a mess we know about.

> because it's used by so many different layers of software without any
> clear guidance about which layers undo which portions of the
> encoding.  So I doubt that it's a good idea to specify a uniform rule
> for how every namespace should encode its native names to URN format;
> instead, I think that encoding considerations should be part of
> namespace-specific "canonicalization" rules.
>
> As for interoperability: if a URN appears with its urn:nid prefix,
> software should be able to assume that it's already in appropriate
> format, with all necessary encoding applied, and apply uniform
> parsing rules, uniform resolution code, etc. to that URN.   But if an
> identifier appears without a urn:nid prefix, software shouldn't be
> trying to generate a URN from it, or otherwise treat it as a URN,
> unless that software knows the namespace-specific rules for encoding
> those identifiers as URNs.   And somewhere, probably in this
> document, there should be a discussion about the difference between
> URN format and namespace-specific "native" formats of identifiers.

I agree that some text about that would be helpful.

> CURRENT:
>
> Depending on the rules governing a namespace, names that are valid
> in a namespace might contain characters that are not allowed by the
> "pchar" production referenced above (e.g., characters outside the
> ASCII range or characters that are reserved in URIs, such as "?",
> and "#").  While such a string might be a valid name, it is not a
> valid URN until it has been translated into a conformant NSS.
> Translation is done by percent-encoding each disallowed character
> using the method defined in Section 2.1 of the generic URI
> specification [RFC3986].  Note that the "%" character is allowed in
> the NSS only for the purpose of percent-encoding.
>
> RECOMMENDED:
>
> Depending on the rules governing a namespace, names that are valid
> in a namespace might contain characters that are not allowed by the
> "pchar" production referenced above (e.g., characters outside the
> ASCII range or characters that are reserved in URIs, such as "?",
> and "#").  While such a string might be a valid name, it is not a
> valid URN until it has been translated into a conformant
> NSS.*Translation****of an identifier from its "native" format to URN
> format is accomplished**by using the canonicalization and encoding
> methods  defined for that **URN namespace in that namespace's
> registration.  Software that is not aware of those URN
> canonicalization**and encoding rules for the namespace used MUST NOT
> construct URNs  from such identifiers.*

That seems fine, with the proviso that I think it might be reasonable to 
recommend percent-encoding.

> *7.* Last paragraph of section 3.2:
>
> CURRENT:
>
> If a namespace designates one or more characters conforming to the
> "pchar" rule as having special meaning for that namespace (e.g.,
> "@") and the namespace also uses that character in a literal sense,
> when used in a literal sense the character MUST be percent-encoded
> (e.g., "%40").  For related considerations with regard to NID
> registration, see Section 7.3.2.
>
>
> RECOMMENDED:
>
> If a namespace designates one or more characters conforming to the
> "pchar" rule as having special meaning for that namespace (e.g.,
> "@") and the namespace also uses that character in a literal
> sense,*that ***    *namespace registration MUST specify a means of
> representing *    *its native format*  *identifiers as URNs in a way
> that conforms *    *with the URN NSS grammar*.*See Section 7.3.2.*

+1

Peter

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 3 Sep 2015 19:38:27 -0600

Third installment...

On 7/1/15 4:03 PM, Keith Moore wrote:

> *8.* Section 3.3:
>
> I have several problems with this section overall:
>
> Defining new URN components without saying anything about semantics
> (a) invites inconsistent interpretation of those components from one
> URN namespace to the next (which thwarts the ability to write client
>  software that handles URNs in a uniform way), and (b) also threatens
> to harm the ability to specify queries and/or fragments with URNs
> that resolve to URLs.
>
> Even though the semantics of q-components and f-components might vary
>  from one URN to the next, it's important for this document to make
> clear that client software can handle these in a uniform manner.  For
> URNs that resolve to URLs, q-component /always/ maps to the URL query
> (if that URL scheme supports query), and f-component /always/ maps to
> the URL fragment.

Syntactically, yes.

> At least for those resources, the semantics of the
>  q-components are defined by those resources, not by the namespace.
> The semantics of f-components are defined by the media type of the
> resource, not by its URN namespace.   (The namespace can choose to
> not assign names to resources that don't fit its criteria for
> interpretation of queries or fragments, but it's important that
> 2141bis not promote the idea that the interpretation of these
> components is arbitrary, or namespace-specific.)

I think I agree. However, it's also important that 2141bis not promote 
the idea that all URNs can be resolved to URLs. Your proviso specifying 
that this behavior applies "for URNs that resolve to URLs" is spot on.

> It's also important to recognize that a resource can quite reasonably
>  have multiple URNs, in the same and/or different namespaces.  This
> must not create a conflict or ambiguity over the interpretation of
> f-components or q-components with respect to that resource, either
> when such components appear within URNs or when those components
> appear in relative references within those resources.

Good point (not that I think it will be a common occurrence for the same 
resource to be identified by URNs in different namespaces).

> If this document is going to focus "almost exclusively" on URN
> syntax, then we need to have an accompanying document that discusses
> semantics of the new components.  I think that's probably a Bad Idea.
> Every time a reader has to switch from one document to another,
> there's a good chance that he'll lose some context, or worse, not
> bother to read the next document.   I'm fine with -semantics-clarif
> being in a separate document if it just explains why certain
> decisions were made, but it's probably not a good idea to entirely
> separate syntax and semantics.   So I'm proceeding on the assumption
> that at least minimum interpretation/semantics of these new
> components needs to be specified in 2141bis.

I guess I'm not as uncomfortable as you are regarding the relative lack 
of detail about the semantics (especially of r-components), but then I 
am not optimistic about the ability of this WG or any other group to 
come to agreement about that for all URN namespaces.

> CURRENT:
>
> Because this specification focuses almost exclusively on URN syntax,
> it does not define the semantics of the q-component, r-component,
> and f-component for URNs in general.  Instead, additional
> specifications might establish these matters for URN-related services
> (such as URN resolution) or for individual URN namespaces (e.g., to
> handle extended information about the resource identified by a URN).
> Defining the details of such usage is the responsibility of
> specifications for URN resolution services, namespace registration
> requests or specifications for individual namespaces, and other
> appropriate documentation (such as policy documents governing the
> management of a given URN namespace).
>
> Unless specifically defined for a particular namespace after
> publication of this document, use of these components is disallowed,
> thereby maintaining strict backward compatibility with namespaces
> defined in accordance with [RFC2141] and registered in accordance
> with [RFC3406].
>
>
> SUGGESTED:
>
> *As compared to [RFC2141], this document extends the URN syntax to***
> *permit inclusion of three new types of components: q-component,***
> *r-component, and f-component. **Because this specification focuses
> ***    *almost exclusively on URN syntax, it does not define detailed
> ***    *semantics for these components for URNs in general. However,
> each of ***    *these components has a distinct role from the others,
> which is***    *independent of the URN and its namespace, and it is
> intended that ***    *clients be able to handle these components
> uniformly for all URNs. ***    *In general: ******o q-component is
> intended for parameters to be transmitted to ****the named resource
> and interpreted by that resource; ******o r-component is intended for
> parameters to be transmitted to ****resolution services and
> interpreted by those services; and ******o f-component is intended to
> be interpreted by the client as ****a specification for a location
> within or region of, the****named resource.*** *The above is true
> regardless of the URN's namespace.******Whenever a URN resolves to a
> URL which may be used to access the****resource, there is a more
> specific interpretation of q-component and ****f-component: the
> q-component is copied verbatim to the query portion****  of the URL
> (if that URL scheme supports query), and the f-component****  is
> copied verbatim to the fragment portion of the URL. This
> is****necessary, among other reasons, so that interpretation of
> q-component****and f-component when associated with URNs, will be
> consistent with****the interpretation of relative references
> containing query and/or ****fragment, respectively, within documents
> which are accessible via URNs.****Thus, for resources which may be
> accessed via a URL, the semantics of ****q-component are identical to
> those for queries to that resource, and ****the semantics of
> f-components are identical to those of fragments for****that
> resource. Semantics of q-component and f-component for
> resources****that inherently cannot be accessed via a URL are
> undefined by this ****document; however, they SHOULD be consistent
> with the above roles. ****It is expected that additional
> specifications will define the semantics****of
> r-components.******Note: In general, neither syntax nor semantics of
> q-component or f-component are**defined by, or dependent on, the
> namespace of the URN. **A particular**namespace might, however,
> define use of r-components that are specific **to its namespace and
> supported by the resolution services  which that**namespace operates
> and/or recommends.****Any of these components MAY be used with URNs
> from existing namespaces whether or not the namespace explicitly
> supports them. As described above, the interpretation of q-component
> and f-component is namespace- independent. Interpretation of an
> r-component may be namespace-dependent to some degree if it relies on
> behavior of a namespace-specific resolution service. If a URN
> contains r-components that are not defined by the namespace, the
> meaning of the URN and result of resolution are out of scope for this
> specification. *

Despite the caveats I expressed above, I think that text is mostly fine. 
I will look it over more carefully before we're done.

> Note about the last paragraph of the suggested text as compared to
> the original: I think it's pointless to forbid use of a *-component
> with existing URNs, especially when the client's interpretation of
> such a component should be consistent from one URN to the next
> anyway.   It's unreasonable to expect clients to implement
> namespace-specific behavior for interpretation of *-components. *

Even for r-components? Or do you say that because we're talking about 
clients here?

> **9.* Section 3.3.1
>
> Again, it is important that this document be clear about the
> appropriate roles for each of these components, and for this document
> to not encourage inconsistent use of these components from one client
>  application or namespace to another.   Also, it's important to be
> clear that q-components are not passed to URN resolvers, so that they
> are not taken into account in URN resolution.

Agreed.

> Also, the example chosen in the -12 text to illustrate use of
> q-components in this section, while valid, seems likely to cause
> confusion.  While it is certainly possible and reasonable to have a
> resource that returns metadata about other resources, and for a URN
> to be assigned to a metadata-returning resource, the example URN
> doesn't actually specify the URI of a resource for which metadata is
> to be returned.   The reader is likely to presume that the resource
> named is returning metadata about itself.  And while a resource could
> certainly do that (and it might be a useful convention), that's
> assuming a lot more than anything that's been specified to date.
> Normally one does not obtain metadata by querying a resource,

Would you consider HTML <head/> to be an exception? Or are you thinking 
about different kinds of metadata?

> one
> obtains metadata by contacting a resolution service - and the
> r-component, not the q-component, must be used to pass paramters to
> the resolution service.
>
> CURRENT:
>
> 3.3.1.  q-component
>
> Although the URN q-component has close to the same syntax as the URI
> query component, the URN q-component need not be semantically
> equivalent to the URI query component and therefore is called by a
> different name in this specification to emphasize that the
> relationship is about syntax only (see also
> [I-D.ietf-urnbis-semantics-clarif]).
>
> The q-component is indicated by the first question mark ("?")
> character and terminated by a double question mark ("??"), by a
> number sign ("#") character, or by the end of the URI.  The q-string
> MUST NOT begin with a question mark ("?") since a double question
> mark is reserved for use by an r-component (Section 3.3.2).  The
> characters slash ("/") and question mark ("?") may represent data
> within the q-string.  Note that characters outside the ASCII range
> [RFC20] MUST be percent-encoded using the method defined in Section
> 2.1 of the generic URI specification [RFC3986].
>
> As described under Section 4, the q-component SHALL NOT be taken
> into account when determining URN equivalence.
>
> The q-component is primarily intended to be used for passing
> information in requests to the resources identified by a URN or to
> applications that manage such resources.  Consider the hypothetical
> example of passing parameters to an application that returns
> metadata about a resource (say, Dublin Core [RFC5013] data about a
> published book).  This could perhaps be accomplished by specifying
> the desired metadata field (e.g., "description") in the q-component,
> resulting in URNs such as
> "urn:example:0-395-36341-1?operation=search&field=description".
> However, this primary purpose is not intended to forestall other
> potential uses for q-components.
>
>
> SUGGESTED:
>
> (note: the following text assumes the grammar alternative 2 and will
> need to be changed if a different fix for the grammar is chosen)

I think alternative 2 is sensible.

> 3.3.1.  q-component
>
> *The URN q-component has the same syntax as the URI query component.
> ****If a URN resolves to a URL with a scheme that supports a query
> ****component, the q-component from the URN is copied verbatim to
> the****query component of the URL.**If the URN does not resolve to a
> URL,****the interpretation of the q-component is undefined by this
> ****specification; however, it SHOULD be used to communicate
> parameters****that are interpreted by a resource. The q-component
> MUST NOT be  used**to communicate parameters to a resolution
> service.********The q-component follows the r-component if one
> exists. The start of the**q-component is indicated by**a**question
> mark ("?") character and ****terminated by a number sign ("#")
> character, or by the end of the URI. * The characters slash ("/") and
> question mark ("?") may represent data within the q-component. Note
> that characters outside the ASCII range [RFC20] MUST be
> percent-encoded using the method defined in Section 2.1 of the
> generic URI specification [RFC3986]. As described under Section 4,
> the q-component SHALL NOT be taken into account when determining URN
> equivalence.*Similarly, the q-component SHALL NOT be taken into
> account when resolving a URN and MUST**NOT be passed  to a resolution
> service when querying the service for metadata.*
>
> The q-component is*entirely*  intended to be used for passing
> information in requests to the resources identified by a URN or to
> applications that manage such resources.*Consider the hypothetical
> example of passing parameters to an application that returns
> weather****maps from different regions and/or for different time
> periods. This****could perhaps be accomplished by specifying lat/long
> coordinates and****dates/times in the URN's q-component.*

That seems mostly fine to me. Again, I'll look at it more carefully 
before we're done.

> * **10. * Section 3.3.2

If we follow alternative 2, this section would move up in the document 
(before the section on q-component).

> CURRENT:
>
> 3.3.2.  r-component
>
> The URN r-component has no syntactic equivalent in URIs.
>
> The r-component is indicated by a double question mark ("??") and
> terminated by a number sign ("#") character or by the end of the
> URI. The characters slash ("/") and question mark ("?") may represent
> data within the r-string.  Note that characters outside the ASCII
> range [RFC20] MUST be percent-encoded using the method defined in
> Section 2.1 of the generic URI specification [RFC3986].
>
> As described under Section 4, the r-component SHALL NOT be taken
> into account when determining URN equivalence.
>
> The r-component is primarily intended to be used for passing
> information in requests to URN resolution services (not, as with the
> q-component, to the resources identified by a URN or to applications
> that manage such resources).  Consider the hypothetical example of
> passing parameters to resolution service (say, an ISO alpha-2
> country code [ISO3166-1] in order to scope down the preferred country
> in which to search for a physical copy of a book).  This could
> perhaps be accomplished by specifying the country code in the
> r-component, resulting in URNs such as
> "urn:example:0-395-36341-1??cc=uk". However, this primary purpose is
> not intended to forestall other potential uses for r-components.
>
> This document defines only the syntax of the r-component and
> reserves it for future use.  The exact semantics of the r-component
> and its use in URN resolution protocols are a matter for potential
> standardization in separate specifications.
>
>
> SUGGESTED:
>
> (note: the following assumes the grammar alternative 2 and will need
> changes for different fixes to the grammar)
>
> 3.3.2.  r-component
>
> The URN r-component has no syntactic equivalent in URIs.
>
> ***The r-component precedes any q-component.

I suppose that would be "any q-component or f-component"?

> It is indicated by a
> double question mark ("??") and terminated by a number sign ("#")
> character, a question mark ("?"), or by the end of the URI. The slash
> ("/") may represent data within the r-component, but the question
> mark ("?") MUST NOT appear in an r-component. If there is a need to
> represent a question mark in data passed to a resolver, it MUST be
> percent-encoded in the URN.*****    Note that characters outside the
> ASCII range [RFC20] MUST be percent-encoded using the method defined
> in Section 2.1 of the generic URI specification [RFC3986].
>
> As described under Section 4, the r-component SHALL NOT be taken
> into account when determining URN equivalence.*The r-component SHALL
> be ****supplied along with the URN when presenting requests to URN
> resolution****services.*
>
> The r-component is*entirely*  intended to be used for passing
> information in requests to URN resolution services (not, as with the
> q-component, to the resources identified by a URN or to applications
> that manage such resources).  Consider the hypothetical example of
> passing parameters to resolution service (say, an ISO alpha-2
> country code [ISO3166-1] in order to scope down the preferred country
> in which to search for a physical copy of a book).  This could
> perhaps be accomplished by specifying the country code in the
> r-component, resulting in URNs such as
> "urn:example:0-395-36341-1??cc=uk".

Per Juha's feedback, that example needs to change.

> This document defines only the syntax of the r-component and
> reserves it for future use.  The exact semantics of the r-component
> and its use in URN resolution protocols are a matter for potential
> standardization in separate specifications.
>
>
> *11.*  Section 3.3.3
>
> CURRENT
>
> 3.3.3.  f-component
>
> Although the URN f-component has the same syntax as the URI fragment
> component, the URN f-component need not be semantically equivalent
> to the URI fragment component and therefore is called by a different
> name in this specification to emphasize that the relationship is
> about syntax only (see also [I-D.ietf-urnbis-semantics-clarif]).
>
> The f-component is indicated by the presence of a number sign ("#")
> character and terminated by the end of the URI.  Note that
> characters outside the ASCII range [RFC20] MUST be percent-encoded
> specification [RFC3986].
> As described under Section 4, the f-component SHALL NOT be taken
> into account when determining URN equivalence.
>
> The f-component is primarily intended to distinguish the integral
> parts of resources named by URNs in particular namespaces.  Consider
> the hypothetical example of obtaining resources that are part of a
> larger entity (say, the chapters of a book).  Each part could be
> specified in the f-component, resulting in URNs such as
> "urn:example:978-952-10-7060-0#chapter1".  However, this primary
> purpose is not intended to forestall other potential uses for
> f-components.
>
>
> ALTERNATIVE
>
> 3.3.3.  f-component
>
> *The URN f-component has the same syntax as the URI fragment
> component. When a URN containing an f-component resolves to a
> URL,****the f-component from the URN is copied verbatim into the
> fragment****of that URL. Interpretation of the f-component for URNs
> that do****not resolve to URLs is not defined by this
> specification.*** The f-component is indicated by the presence of a
> number sign ("#") character and terminated by the end of the URI.
> Note that characters outside the ASCII range [RFC20] MUST be
> percent-encoded using the method defined in Section 2.1 of the
> generic URI specification [RFC3986].
>
> As described under Section 4, the f-component SHALL NOT be taken
> into account when determining URN equivalence.*Similarly, the
> f-component****MUST NOT be passed to resolution servers when querying
> them for resource**locations or metadata.*
>
> The f-component is primarily intended to distinguish the integral
> parts of resources named by URNs in particular namespaces.  Consider
> the hypothetical example of obtaining resources that are part of a
> larger entity (say, the chapters of a book).  Each part could be
> specified in the f-component, resulting in URNs such as
> "urn:example:978-952-10-7060-0#chapter1".  However, this primary
> purpose is not intended to forestall other potential uses for
> f-components*for URNs that do not resolve to
> network-accessible****resources. *

Here again that seems mostly fine to me.

Thanks for all the suggested text!

Peter

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 3 Sep 2015 19:40:30 -0600

Fourth installment...

On 7/1/15 4:03 PM, Keith Moore wrote:

> *12.*  Section 5.3
>
> (NIT) I think it's a bit of a stretch to say what the result of use of a
> relative
> reference will be if there's no base URI.
>
> CURRENT:
>
>     If no such base URI exists, use of a relative reference with respect
>     to a URN is an error and will yield results that might be unexpected
>     or not useful.
>
> RECOMMENDED:
>
>     If no such base URI exists, use of a relative reference with respect
>     to a URN is an error.*Client behavior in this case is undefined. *

WFM.

Peter

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 3 Sep 2015 19:46:10 -0600

Fifth installment...

On 7/1/15 4:03 PM, Keith Moore wrote:

> *13.*  Section 6, about halfway through
>
> Clarify that a resource MAY have more than one URN assigned to
> it, either in the same namespace or in different namespaces.
>
> CURRENT:
>
>     With regard to global uniqueness, using different NIDs for different
>     collections of identifiers ensures that no two URNs will be the same
>     for different resources, since each collection is required to
>     uniquely assign each identifier.  However, a single resource can have
>     more than one URN assigned to it for different purposes (for example,
>     if a book were published in a monograph series, it could have both an
>     ISBN [RFC3187] and an ISSN [RFC3044] assigned to it, resulting in two
>     URNs referring to the same book).  Subject to other constraints, such
>     as those imposed by the URI syntax [RFC3986], the rules of the URN
>     scheme are intended to allow preserving the normal and natural form
>     of identifiers specified elsewhere when they are treated as URN
>     namespaces.
>
>
> RECOMMENDED:
>
>     With regard to global uniqueness, using different NIDs for different
>     collections of identifiers ensures that no two URNs will be the same
>     for different resources, since each collection is required to
>     uniquely assign each identifier.*However, a single resource MAY have more than one URN assigned to it,
> either in the same namespace (if the namespace permits it) or**in different namespaces, and either for similar purposes or different **purposes.*  (for example,
>     if a book were published in a monograph series, it could have both an
>     ISBN [RFC3187] and an ISSN [RFC3044] assigned to it, resulting in two
>     URNs referring to the same book).  Subject to other constraints, such
>     as those imposed by the URI syntax [RFC3986], the rules of the URN scheme
>     are intended to allow preserving the normal and natural form of
>     identifiers specified  elsewhere when they are treated as URN
>     namespaces.

That's an improvement, thanks.

> *14.* section 6, last paragraph
>
> If we're deprecating X- namespaces, fine, but we should reserve
> namespaces beginning with X- to prevent conflicts with identifiers
> that might already exist in the wild.
>
> CURRENT:
>
>     Note: [RFC3406] defined a third type of "experimental namespaces",
>     denoted by prefixing the namespace identifier with the string "X-".
>     Consistent with general IETF conclusions about similar approaches
>     [RFC6648], this specification removes the experimental category and
>     syntax.  Because experimental namespaces were never registered,
>     removing the experimental category has no impact on the existing
>     registries or future registration procedures.  Because they are not
>     registered, strings that refer to existing experimental namespaces
>     are not valid URNs.  Truly experimental usages can, of course, employ
>     the 'example' namespace [RFC6963].
>
>
> RECOMMENDED:
>
>     Note: [RFC3406] defined a third type of "experimental namespaces",
>     denoted by prefixing the namespace identifier with the string "X-".
>     Consistent with general IETF conclusions about similar approaches
>     [RFC6648], this specification removes the experimental category and
>     syntax.  Because experimental namespaces were never registered,
>     removing the experimental category has no impact on the existing
>     registries.*However, to avoid the potential for conflict between****previously-allowed unregistered experimental namespaces and ****namespaces registered in the future, no registrations will be****accepted for new namespaces beginning with "X-". *Because they are not
>     registered, strings that refer to experimental namespaces
>     are not valid URNs.  Truly experimental usages*MAY*, of course, employ
>     the 'example' namespace [RFC6963].

Yes, that's better.

> *15.* Section 6.1
>
> CURRENT:
>
>     5.  It MUST NOT start with the string "xn--" or any other string
>         consisting of two letters followed by two hyphens.  Those strings
>         are reserved for potential representation of DNS A-labels and
>         similar strings in the future [RFC5890].
>
> RECOMMENDED:
>
>     5.  It MUST NOT start with the string "xn--" or any other string
>         consisting of two letters followed by two hyphens.  Those strings
>         are reserved for potential representation of DNS A-labels and
>         similar strings in the future [RFC5890].
>
> *6. It MUST NOT start with the string "X-" so that it will not be
> confused with an experimental namespace previously permitted by
> [RFC3406], nor conflict**with such a namespace.*

+1.

Peter

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 3 Sep 2015 20:45:48 -0600

On 9/3/15 5:53 PM, Peter Saint-Andre - &yet wrote:
> Although I have no objections to including such text,

Here is proposed text (adding the second paragraph below):

    This document modifies the syntax of the NSS to allow the following
    characters: "/", "~", and "&".  In particular, allowing the "/"
    character effectively makes it possible to encapsulate hierarchical
    identifiers from other naming systems.  For instance, consider the
    hypothetical example of a hierarchical naming system in which the
    identifiers take the form of a series of numbers separated by the "/"
    character, such as "1/406/47452/2".  If the naming authority for such
    identifiers were to use URNs, it would be natural to place the
    existing identifiers in the NSS, resulting in URNs such as
    "urn:example:1/406/47452/2".

    However, the foregoing changes to the syntax for the NSS do not
    change the encoding rules for URN namespaces that were defined in
    accordance with [RFC2141].  If any existing URN namespace that is
    used outside of the URN context (i.e., as a standalone identifier
    space) also allows the use of "/", "~", or "&" in the native form
    within that identifier space, then the encoding rules for that
    namespace are not changed by this specification.

> the universe of

I propose adding a new sentence to the very first paragraph of the 
introduction:

    A Uniform Resource Name (URN) is a Uniform Resource Identifier (URI)
    [RFC3986] that is assigned under the "urn" scheme and a particular
    namespace, with the intent that the URN will be either a persistent,
    location-independent resource identifier or in some cases an abstract
    designator that is persistent but that does not identify a resource.
    A URN namespace is a collection of such identifiers, each of which is
    (1) unique, (2) assigned in a consistent and managed way, and (3)
    assigned according to a common definition.  (Some URN namespaces
    create names that exist only as URNs, whereas others create URNs out
    of names that already exist in other identifier systems, such as
    ISBNs [RFC3187] and ISSNs [RFC3044].)

> recommend percent-encoding.

I propose the following adjusted text:

    Depending on the rules governing a namespace, names that are valid in
    a namespace might contain characters that are not allowed by the
    "pchar" production referenced above (e.g., characters outside the
    ASCII range or characters that are reserved in URIs, such as "?", and
    "#").  While such a string might be a valid name, it is not a valid
    URN until it has been translated into a conformant NSS.  In the case
    of URNs that are formed from names that exist separately in a
    standalone identifier space, translation of an identifier from its
    "native" format to URN format is accomplished by using the
    canonicalization and encoding methods defined for that URN namespace.
    Such methods SHOULD use percent-encoding unless the standalone
    identifier space already defines different encoding rules for such
    characters within the native format.  Software that is not aware of
    those namespace-specific canonicalization and encoding rules MUST NOT
    construct URNs from the names in the standalone identifier space.

>> that conforms *    *with the URN NSS grammar*.*See Section 7.3.2.*

I propose the following adjusted text:

    If a namespace designates one or more characters conforming to the
    "pchar" rule as having special meaning for that namespace (e.g., "@")
    and the namespace also uses that character in a literal sense, when
    used in a literal sense the namespace registration MUST specify a
    method of representing its native-format identifiers as URNs in a way
    that conforms with the generic URN NSS grammar.  This method SHOULD
    use percent-encoding unless the standalone identifier space already
    defines different encoding rules for such characters.  For related
    considerations with regard to NID registration, see Section 7.3.2.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: "Svensson, Lars" <[email-masked]>
Date: Fri, 4 Sep 2015 14:54:54 +0000

> character of a q-component to be "?".

Yes, but only if there is an r-component in the urn. If you only have

urn:example:foo??bar=baz

you cannot determine if the last nine characters should be interpreted as an r-component "bar=baz" or a q-component "?bar=baz". It seems that we still cannot allow the first character of the q-component to be '?' unless we choose another character to delimit the r-component.

Or am I mistaken?

Best,

Lars

---

From: Keith Moore <[email-masked]>
Date: Fri, 4 Sep 2015 12:15:10 -0400

> you cannot determine if the last nine characters should be interpreted as an r-component "bar=baz" or a q-component "?bar=baz". It seems that we still cannot allow the first character of the q-comp…
I believe you are correct.

---

From: Keith Moore <[email-masked]>
Date: Fri, 4 Sep 2015 13:41:07 -0400

>    namespace are not changed by this specification.

Mostly looks good, though I think I would change the first sentence to 
say "... do not change the encoding rules for existing URN namespaces" 
because a reader doesn't have a good way to know whether a namespace 
registration was defined according to 2141 or not.

>    ISBNs [RFC3187] and ISSNs [RFC3044].)

Looks good.   (I assume those are informative references?   I have heard 
that they're likely to be updated soon and would hate for those updates 
to delay adoption of 2141bis.)

>> situations where the creativity of the namespace registrant could lead

I don't think namespace registrants should be constrained to use percent 
encoding if there's a more readable and/or transcribable way to convert 
their names from native format to URN NSS format (say, by substituting 
one separator character for another).   I also think that (for similar 
reasons) imposing percent encoding as a constraint could discourage some 
namespaces from registering as URNs.   Perhaps that isn't a tragedy, 
since we probably can't expect that all persistent namespaces should be 
URNs anyway, but it seems unfortunate.

I also think it's very important that client software not assume that 
the way to translate native format names to URNs is to percent-encode 
them.   So encouarging the percent-encoding convention for this, or 
making it a SHOULD, actually seems harmful to me.
>    construct URNs from the names in the standalone identifier space.

I would remove the penultimate sentence, but otherwise fine.

>    considerations with regard to NID registration, see Section 7.3.2.

As above, I would remove the penultimate sentence.

Keith

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Sep 2015 13:59:43 -0400

--On Friday, September 04, 2015 14:54 +0000 "Svensson, Lars"
<[email-masked]> wrote:

> you cannot determine if the last nine characters should be
> interpreted as an r-component "bar=baz" or a q-component
> "?bar=baz". It seems that we still cannot allow the first
> character of the q-component to be '?' unless we choose
> another character to delimit the r-component.
> 
> Or am I mistaken?

I believe you are correct... and this was one of the
considerations that led to the initial proposal/ decision to put
q-component first, after which one could make 3986-compatible
syntax restrictions and interpretation on what came within what
3986 would continue to consider a query but URNs would consider
a q-component followed by an r-component.

The explanation that follows is an abbreviated version of the
one I've been resisting trying to reconstruct, partially because
I haven't looked at, or had occasion to apply except via tools
that hide it, the theory involved carefully in some years and
can't be sure I've got the explanation below completely right.
Even then, I'm assuming that everyone in the WG understands LR
and LALR parsing at a level beyond what appears in tutorials for
various large bovine parser generators.  FWIW, I'm a lot less
comfortable trying to simulate a computer scientist at this
point in my career than pretending to be a network engineer, but
maybe the following will help albeit with apologies to those who
don't know the literature.

Consider the hypothetical, but often cited, generic URI parser
that breaks things into components with your string above and
with 

urn:example:foo?bar=baz
urn:example:foo??bar=baz?aaa?bbb=cdd

Doing this from memory, so excuse small errors, the generic URI
parse trees then look like

	Scheme: urn
	Hier-part: example:foo
	Query: ?bar=baz

	Scheme: urn
	Hier-part: example:foo
	Query: bar=baz

and

	Scheme: urn
	Hier-part: example:foo
	Query: ?bar=baz?aaa??bbb=cdd
	
Now, in all three cases, the Query is perfectly valid under
3986.  It is under the second too, but any ideas of being able
to pass r-component off to a separate processor without passing
through the query processor first essentially disappear.  Even
in the third case a URN-specific query processor cannot tell
unambiguously whether the r-component is "bbb=cdd" or the
entirety of "bar=baz?aaa??bbb=cdd".

Now, if r-component follows the q-component, then, while the
3986 syntax remains 

	scheme ":" hier-part [ "?" query ]

it is easy to imagine a URN-specific parser applied to the tree
the generic URI parser produces.   That parser, or at least part
of it, is needed anyway to separate NID from NSS.

So, for the "r-component after" model, the generic URN parser,
applied to the trees above, produces

	Scheme: urn
	NID: example
	NSS: foo
	q-component: ?bar=baz
	r-component: (empty)
       (because, at least as I first spelled it out, the
introducer for an r-component is two consecutive "??" _inside_
the query.  If one wanted

	q-component: (empty)
	r-component: bar=baz 

then one would need to write 

	urn:example:foo???bar=baz

which is an eyesore, but at least unambiguous.  The third case
is clear:

	Scheme: urn
	Hier-part: example:foo
	q-component: ?bar=baz?aaa
	r-component: bbb=cdd

One can decide that there is really no such thing as a generic
URI parser (at beyond separation of scheme from tail) and recast
things in LR terms.  But, that way, the "r-component second"
model requires only an LALR(1) engine because one needs to look
ahead only to see if another "?" (or a "#" is coming.  But
dealing with figuring out whether an r-component is actually
present in the "r-component before" case requires either
prohibiting any use of un-encoded "?" within the q-component or
an LR(k) engine for essentially arbitrary _k_ to determine
whether any question marks appear in the query and, if so,
whether it is doubled.

There are a number of parsing scenarios that don't raise any of
the above issues and that make the choice between "r-component
before" and "r-component after" either arbitrary or a matter of
taste.  But I have lost track of how many times the WG has been
told that there are generic URI parsers out there and that we
must design URNs so that they work.  I can find nothing in 3986
that prohibits such a parser from breaking the URI into
substrings (or tree nodes) and then passing those subtrees off
to action routine that do not have the ability to reconstruct
and then re-parse the whole URI string.

best,
   john

    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Sep 2015 14:07:57 -0400

--On Friday, September 04, 2015 13:41 -0400 Keith Moore
<[email-masked]> wrote:

> (Some URN namespaces
>>    create names that exist only as URNs, whereas others
>>    create URNs out of names that already exist in other
>>    identifier systems, such as ISBNs [RFC3187] and ISSNs
>>    [RFC3044].)
> 
> Looks good.   (I assume those are informative references?   I
> have heard that they're likely to be updated soon and would
> hate for those updates to delay adoption of 2141bis.)

They certainly look like informative references to me.  More to
the point, draft-ietf-urnbis-ns-reg-transition, also in the WG's
queue, discusses the fate of those documents and their
relationship to the URNBIS effort.   If you think it would be
helpful, we couple presumably incorporate a reference to it in
2141bis.  While it seems obvious that we need to get 2141bis
locked down before giving significant attention to the other two
pending specs (draft-ietf-urnbis-semantics-clarif being the
other one), I cannot imagine a scenario in which we did not try
to progress all three together.

    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Sep 2015 14:19:02 -0400

--On Thursday, September 03, 2015 20:45 -0600 Peter Saint-Andre
- &yet <[email-masked]> wrote:

>     "#").

Peter,

I understand and appreciate the desire to be helpful, but please
be careful here because "reserved in URIs" is a tricky concept,
with some characters that are candidates for reservation within
schemes and some that are always reserved except when they
appear in specific context, such as left-to-right after an
appearance of the same character.   I'd hate to see us add to
the confusion.

Similarly,

> translation of an identifier from its
>     canonicalization and encoding methods defined for that URN
> namespace.     Such methods SHOULD use percent-encoding unless
> the standalone     identifier space already defines different
> encoding rules for such     characters within the native
> format.  Software that is not aware of     those
> namespace-specific canonicalization and encoding rules MUST NOT

I got a bad headache reading that out of context the first
couple of times.  If you are talking about the methods, they
don't "user percent-encoding", they interpret strings containing
percent signs as percent-encoding as, in turn, specified in RFC
3986 rather than interpreting the percent signs in some other
way (or not at all).

> If a namespace designates one or more characters conforming to
> the     "pchar" rule as having special meaning for that
> namespace (e.g., "@")     and the namespace also uses that
> character in a literal sense, when     used in a literal sense
> the namespace registration MUST specify a     method of
> representing its native-format identifiers as URNs in a way
> that conforms with the generic URN NSS grammar.

Is a sentence I could have written, one of the ones people claim
is incomprehensible on first (or even third or fourth) reading.
If you need it, please break it up into pieces.

I will probably have more to say about this (and maybe some more
specific suggestions) when I can see things in more context in
-13.

     john

    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Sep 2015 14:25:52 -0400

--On Tuesday, September 01, 2015 21:20 -0600 Peter Saint-Andre -
&yet <[email-masked]> wrote:

>> b) The grammar refers to "q-string" etc. but the text refers
>> to "q-component" etc.  I could find no reason for the
>> discrepancy.
> 

A bit more than that.   At one time, a distinction was made with
grammar that looked more like

     namestring  = assigned-name [ q-string ] ...
     q-string = "?" q-component
     ...

I don't remember whether I was worried about some difficulty
with clarity or if it was tied up with the treatment of "/" and
the former p-component idea.   In any event, when that was
simplified, I obviously didn't correctly realign it with the
text.   Mea Culpa... sorry.

Definitely safe to fix.

    john

---

From: Keith Moore <[email-masked]>
Date: Fri, 4 Sep 2015 19:42:41 -0400

On 09/04/2015 01:59 PM, John C Klensin wrote:
> I believe you are correct...

mostly so.   The "alternative 2" grammar I suggested is ambiguous 
because the parser doesn't know whether to reduce

'?' '?' pchar  -> '?' q-component

or

'?' '?' pchar -> r-component

and no amount of lookahead will resolve this conflict unless there is an 
additional '?' in the URN.

What this would mean in practice is that absent some explicit rule to 
resolve the ambiguity, different correct implementations might parse the 
same URNs in different ways, which would be bad.

> can't be sure I've got the explanation below completely right.

Your explanation is basically correct as far as I can tell, though I 
didn't go through it with a fine-toothed comb.   However, IMO it's 
really not necessary for a URN parsing grammar to be LALR(1) or to limit 
lookahead to one symbol.   I expect that hardly anyone is going to use a 
parser generator to construct a URN parser, as this is easily 
accomplished with (say) regular expression libraries that are available 
in most modern programming languages.    Such libraries will do as much 
lookahead as needed.

Or if we resolved the ambiguity somehow (e.g. by saying that '?' may 
never appear as the first character of a query without percent-encoding 
it) an LALR(1) parser could parse it because these generally have some 
way to explicitly resolve reduce/reduce conflicts.

So the real problem with alternative 2 isn't the amount of lookahead 
required, it's the ambiguity in the grammar.   But at this point I'm not 
sure that alternative 2 (even with the ambiguity fixed) offers any 
advantage over alternative 1.

As for 3986 compatibility, I think we need to constrain URNs so that 
they are accepted as valid URNs by the 3986 grammar, but I don't think 
we need to worry too much about being compatible with how a 3986 parser 
breaks such a URN down into individual components.   Any software that 
actually tries to resolve URNs is going to need to know how to parse 
them according to 2141bis.

Keith

---

From: John C Klensin <[email-masked]>
Date: Sat, 05 Sep 2015 06:13:08 -0400

Keith,

I think your comments identify the very essence of the
disagreement we are having and the tradeoff choices the WG needs
to make.  That, AFIAC, is great progress.

Inline below.

--On Friday, September 04, 2015 19:42 -0400 Keith Moore
<[email-masked]> wrote:

> Your explanation is basically correct as far as I can tell,
> though I didn't go through it with a fine-toothed comb.
> However, IMO it's really not necessary for a URN parsing
> grammar to be LALR(1) or to limit lookahead to one symbol.   I
> expect that hardly anyone is going to use a parser generator
> to construct a URN parser, as this is easily accomplished with
> (say) regular expression libraries that are available in most
> modern programming languages.    Such libraries will do as
> much lookahead as needed.

I learned a lesson about IT Standards and implementations many
years ago, when the Internet was in its infancy.  A standard was
under development that was, necessarily, hundreds of pages long
and involved a three-level grammar and other sources of
complexity.  For various reasons, some similar to the reasons
XML2RFC has become popular in the IETF, the spec was written is
SGML.   There were GML and SGML libraries around, but, because
of SGML's flexibility, they were complex to use and took effort
to understand.  The folks on the standards development committee
were, however, almost all compiler experts.  The prediction was
that the committee either would rely on the editor for
compilation and work with the output text or would learn and use
those libraries.  I certainly made that prediction and was part
of the second group.  But it didn't work out that way: we ended
up with nearly as many implementations as there were members of
the committee.  The vast majority of them were not SGML
implementations, they were specific to the DTD and actions
required for the particular grammar and conventions needed for
the standard.  The reason?  Typically "we had the right tools
for building something like that in our compiler kits, they were
familiar and easy to apply, and we didn't have to learn a new
and complex package/ library".  Given the complexity of
multi-pass compiler-building toolsets, especially at the time,
that decision is both bizarre and completely rational.

The lesson: You may look at a spec and say "this will mostly be
implemented in regular expressions".  I might even agree.  But,
unless the standard is written with a requirement to implement
in a particular way, it is important to consider _all_ possible
rational implementation models and to understand the loss
functions associated with assuming one of them.

> Or if we resolved the ambiguity somehow (e.g. by saying that
> '?' may never appear as the first character of a query without
> percent-encoding it) an LALR(1) parser could parse it because
> these generally have some way to explicitly resolve
> reduce/reduce conflicts.

Oddly, this is another example of the above.  I recommend
DeRemer's dissertation and the DeRemer-Penello paper as
recreational reading.  If there are conflicts to be resolved,
either the grammar is LR(k) rather than LALR(k) (especially with
k= 0 or 1) or there are deficiencies or ambiguities in the
grammar that need to be resolved in other ways.  That doesn't
imply that no implementation has features to prioritize
constructions; several do.   However, the implementations aren't
the standard (or, in this case, the abstract model).

>...
> As for 3986 compatibility, I think we need to constrain URNs
> so that they are accepted as valid URNs by the 3986 grammar,
> but I don't think we need to worry too much about being
> compatible with how a 3986 parser breaks such a URN down into
> individual components.   Any software that actually tries to
> resolve URNs is going to need to know how to parse them
> according to 2141bis.

Well, I've believed all along that, in practice, all the
hypothetical generic URI parser is going to do is the most
superficial of pattern-conformance checks (if that) and then
pass the entire URI (or everything other than the scheme name)
string  to a scheme-specific parser.  That belief, along with
listening to many discussions about what real implementations do
in the wild, is why I keep putting "hypothetical" in front of
"generic URI parser".  That is completely consistent with the
above.   However, assume instead that the generic parser
actually breaks things up into the pieces specified in Section 3
of RFC 3096 and that it then either passes individual pieces off
to different action routines or calls on a scheme-specific
mechanism with separate pre-processed (e.g., to get rid of
"%"-encoding) elements [1].  We've been told that there are
implementations that do that.  Any implementation that handles
fragments in the scheme-independent way to which we've been told
URNs must conform has at least some properties of that approach.
In that model, there is still a 2141bis-specific processor, but
it never sees the original URI string, much less has a chance to
parse/interpret the component separation differently than 3986
does.

That, btw, is why I've tended to want to write the 2141bis
grammar along the lines of 

   URN = NID ":" NSS [ "?" query ] [ "#" f-component]
   query = [ q-component "??" r-component ]
     (or some r-component-before variation)

so that the 3986 syntax is clearly intact.  But the way in which
the ABNF is written is a lot less important than what we want
the syntax to be and whether this is compatible with
3986-conformant implementations.[2] 

best,
   john

[1] Many readings of 3986 and the proposed text for 2141bis
assume that the decision to interpret "%" as an encoding
indicator for a specific encoding can be per-scheme or even
per-NID.  As I understand it, WHATWG and at least some of the
IRI folks believe that interpretation is inherent to URIs and is
inescapable.  I can only note that the former group believes
that the definition of "generic URI parser" is "what browsers
do" and that, if we violate that definition, whatever we are
doing won't work in practice regardless of how we read 3986.

[2] I do not advocate going back to it, but this whole mess is
the reason why my initial instinct was "URNs are not URIs" -- a
decision that would, in essence, have acknowledged that there
was no such things as a generic URI parser beyond separation of
scheme from an opaque string and that 3986 was a guide to URI
scheme design, not a parsing or interpretation specification.
In that model, one really could say "URNs get parsed with a
2141bis-specific parser without interference from anything
URI-generic".  But, as long as we require that URNs be strictly
conformant to URI syntax as defined in 3986 and not get
crosswise with any possible 3986-conformant generic parsing
model, we are stuck with this situation.

---

From: Keith Moore <[email-masked]>
Date: Sat, 5 Sep 2015 07:19:12 -0400

> to make.  That, AFIAC, is great progress.

I wasn't aware that we were still having a substantive disagreement 
about URN syntax choices, since I explicitly stated that I no longer see 
an advantage in alternative 2.   So I think that either we need to go 
with alternative 1 (despite its warts) or find another character to 
introduce r-components.   And I realize that there may be no characters 
available for that purpose that are compatible with 3986.

---

From: John C Klensin <[email-masked]>
Date: Sat, 05 Sep 2015 08:09:45 -0400

--On Saturday, September 05, 2015 07:19 -0400 Keith Moore
<[email-masked]> wrote:

>> disagreement we are having and the tradeoff choices the WG
>> needs to make.  That, AFIAC, is great progress.
> 
> I wasn't aware that we were still having a substantive
> disagreement about URN syntax choices, since I explicitly
> stated that I no longer see an advantage in alternative 2.
> So I think that either we need to go with alternative 1
> (despite its warts) or find another character to introduce
> r-components.   And I realize that there may be no characters

Philosophical disagreement, not a substantive one -- I've been
having trouble understanding you and considered that a problem
-- more or one if you were having equal trouble understanding
me.   

Back to Peter... and, fwiw, another of my objections to 3986 is
that it seems to have used up every plausible candidate
single-character delimiter.

best,
   john

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 10 Sep 2015 19:15:13 -0600

> registration was defined according to 2141 or not.

I don't see much effective difference between "existing URN namespaces 
at the time of writing" and "namespaces registered before 2141bis is 
published as an RFC". However, I suppose that even a namespace 
registered after 2141bis is published an RFC could be defined in 
accordance with RFC 2141 syntax (e.g., perhaps it was defined years ago 
and only now registered). I think the pointer to 2141 syntax is more 
precise.

> to delay adoption of 2141bis.)

They sure look informative to me.

> one separator character for another).

That's what SHOULD means here, I think.

>  I also think that (for similar

Well this isn't a hill for me to die on so dropping the recommendation 
is fine with me.

> I would remove the penultimate sentence, but otherwise fine.

Sure.

> As above, I would remove the penultimate sentence.

Fine with me.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 10 Sep 2015 19:33:00 -0600

> appearance of the same character.   I'd hate to see us add to

I see your point. How about:

OLD

    or characters that are reserved in URIs, such as "?", and "#").

NEW

    or other disallowed characters such as "?" and "#").

We could even say:

    or the characters "/", "?", "#", "[", and "]").

> 3986 rather than interpreting the percent signs in some other

Keith suggested we drop that sentence entirely, which I'd find acceptable.

> If you need it, please break it up into pieces.

I am not sure that we need it. In particular, I am not confident that I 
understand what RFC 2141 intended by the terms "special meaning" and 
"literal sense" (see Section 2.3.1 there):

    Namespaces MAY designate one or more characters from the URN
    character set as having special meaning for that namespace.  If the
    namespace also uses that character in a literal sense as well, the
    character used in a literal sense MUST be encoded with "%" followed
    by the hexadecimal representation of that octet.  Further, a
    character MUST NOT be "%"-encoded if the character is not a reserved
    character.  Therefore, the process of registering a namespace
    identifier shall include publication of a definition of which
    characters have a special meaning to that namespace.

> specific suggestions) when I can see things in more context in

OK. I will keep plugging away so that we can publish -13. As a colleague 
of mine says, "ship and iterate".

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 10 Sep 2015 20:00:26 -0600

On 9/5/15 6:09 AM, John C Klensin wrote:
> Back to Peter...

Thanks!

If we've decided to go with Keith's alternative 1, then your suggestion 
of ABNF along the following lines does seem reasonable:

I'll see about adjusting the text along those lines...

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: John C Klensin <[email-masked]>
Date: Fri, 11 Sep 2015 01:31:17 -0400

--On Thursday, September 10, 2015 19:33 -0600 Peter Saint-Andre
- &yet <[email-masked]> wrote:

>...
>> Peter,
>> 
>> I understand and appreciate the desire to be helpful, but
>> please be careful here because "reserved in URIs" is a tricky
>> concept, with some characters that are candidates for
>> reservation within schemes and some that are always reserved
>> except when they appear in specific context, such as
>> left-to-right after an appearance of the same character.
>> I'd hate to see us add to the confusion.
> 

I don't think that helps.  It might make things worse because it
doesn't provide a citation for "diaallowed characters".

>     or the characters "/", "?", "#", "[", and "]").

Perhaps:

	or the characters "/", "?", "#", "[", and "]",
	consistent with the restrictions in RFC 3986).

Not wonderful, but it lets us off the hook, avoids claims about
"reserved" or "disallowed", and points the finger squarely where
it belongs.

>>>      "native" format to URN format is accomplished by using
>>>      the canonicalization and encoding methods defined for
>>>      that URN namespace.     Such methods SHOULD use
>>> percent-encoding unless the standalone     identifier space
>>> already defines different encoding rules for such
>>> characters within the native format.  Software that is not
>>> aware of     those
>>> namespace-specific canonicalization and encoding rules MUST
>>> NOT construct URNs from the names in the standalone
>>>      identifier space.
>> 
>> don't "user percent-encoding", they interpret strings
>> containing percent signs as percent-encoding as, in turn,
>> specified in RFC 3986 rather than interpreting the percent
>> signs in some other way (or not at all).
> 

I do too.

>>> If a namespace designates one or more characters conforming
>>> to the     "pchar" rule as having special meaning for that
>>> character in a literal sense, when     used in a literal
>>> sense the namespace registration MUST specify a     method of
>> Is a sentence I could have written, one of the ones people
>> claim is incomprehensible on first (or even third or fourth)
>> reading. If you need it, please break it up into pieces.
> 
> I am not sure that we need it. In particular, I am not
> confident that I understand what RFC 2141 intended by the
> terms "special meaning" and "literal sense" (see Section 2.3.1
> there):
> 

At least in some readings, earlier versions of URI/URL syntax
specs allowed more per-scheme syntax variations and restrictions
than many readers of RFC 3986 believe it allows.  If we can just
drop the comment and avoid being enmeshed in that discussion,
that would probably be the best solution.  While I dislike doing
this to readers in principle --especially given how hard 3986 is
to follow in places-- I think our decision to stick with 3986
syntax and some of the associated controversy suggests that we
should remove as much text as possible from 2141bis that merely
restates 3986 provisions or syntax unless we actually
need/intend to update it.

>> I will probably have more to say about this (and maybe some
>> more specific suggestions) when I can see things in more
>> context in -13.
> 
> OK. I will keep plugging away so that we can publish -13. As a
> colleague of mine says, "ship and iterate".

:-)
thanks,
   john

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Fri, 11 Sep 2015 06:53:30 -0600

WFM.

> need/intend to update it.

That seems eminently reasonable.

Peter