Subject: Re: Comments on draft-ietf-6man-rfc6874bis-00.txt

From: "Shang Ye" <[email-masked]>
Date: Mon, 4 Apr 2022 20:21:15 +0800

Brian,

Thanks for your enlightening reply! I read about some previous debates on this matter (about the bare "%") and tried to think it through. My points in detail:

First of all, I must say I mostly agree with Andrew Cady on this matter, but I feel quite necessary to point out the idea that underlies Andrew's arguments:

An update to a standard is subject only to practicality (backward compatibility, interoperability, etc.), not constrained by the original standard text.

There seemed to be a lot of misunderstandings about this. In the URI's case, this means that any extension to the URI syntax would be OK as long as the extended syntax ensures interoperability ("extension" means that it's already backward compatible). No any statement in RFC3986 would be stopping this extension, because if there were one we may well update it.

To be more specific:

We're going to update RFC3986 to allow a bare "%" that delimits an optional zone identifier from an IPv6 address. It's practical to do so because this update is a syntax extension (backward compatible with RFC3986) that ensures interoperability (e.g., copy & paste), so this update would be OK.

Below are some comments in line:

On Sat, 26 March 2022 20:11 UTC, Brian E Carpenter wrote:
>> Section 3 says:
>>
>>> According to the text in Section 2.4 of [RFC3986], "%" must
>>> be percent-encoded as "%25" to be used as data within a URI.
>>> However, in the formal ABNF syntax of RFC 3986, this only applies
>>> where the "pct-encoded" element appears.  For this reason, it is
>>> possible to extend the ABNF such that the scoped address
>>> fe80::abcd%en1 would appear in a URI as http://[fe80::abcd%en1] or
>>> https://[fe80::abcd%en1].
>>
>> The logic of this part seems a bit confusing. I guess the main problem here is whether we should interpret the "%" in an IPv6 scoped address as *data* within a URI. If we do so, then I'm afraid we wouldn't get the logic right here.
>
> I find the text in RFC3986 very confusing. When we looked at it ten years ago, we naturally interpreted it as the ABNF in RFC6874 defines, i.e. requiring percent-encoding.

The text in RFC3986 could be confusing, but I believe it's not confusing *in itself*. At the time it was written the intention was clear: "%" is an escape only (although not directly put) and data *is* data. The problem arises *only* when we try to give "%" a second meaning, which confuses people if they hold the misbelief that the text in RFC3986 is *constraining* its update. *Even if* RFC3986 stated clearly that "%" is an escape only, we may well update this statement to say we allow a "%" in the IP literal as a delimiter, just because our update is backward compatible (such presence of "%" is not at all allowed by RFC3986).

> Andrew Cady argued the opposite a few months ago, and suggested the revised ABNF that is now in RFC6874bis.

The revised ABNF now in RFC6874bis is just great, but the text in it still needs some fixing. I'd like to quote some words from Andrew first:

On Tue, 06 July 2021 20:05 UTC, Andrew Cady wrote:
> By the way, there is a relevant mistake of fact in rfc6874:
>
>     According to URI syntax [RFC3986], "%" is always treated as an
>     escape character in a URI,
>
> The "always" part is untrue.  It is only treated that way where BNF says
> "pct-encoded".  Inside of square brackets it must instead be treated as
> an error.  Pct-encoded data is not allowed there.

I don't quite agree with Andrew here, but I believe the real mistake in RFC6874 is that it fails to recognise its exemption from the text in RFC3986. I'm just badly in hope that this mistake will *not* continue into RFC6874bis in any way.

> If the URI parsing obeys the ABNF, either would be OK and we can choose the one we like best.
>
> When I looked at the WGET parser, which is relatively simple to understand, the first thing I realised is that it is *not* following the ABNF in any automated way (like a syntax-driven compiler would, for example). I also looked at the way Firefox parses URLs, which is very hard to understand, but the same is true. From feedback we (the authors) have received, I believe that all browsers are similar: the parsers do what the programmers make them do, rather than mathematically following the ABNF. Further, the WHATWG specifies the agreed behavior in plain English, not in ABNF (https://url.spec.whatwg.org/#host-representation). That is what the browser implementers follow.
>
> So, those confusing words in RFC3986 are just... confusing.
>
> The real-world description of percent-encoding is at https://url.spec.whatwg.org/#percent-encoded-bytes`

But isn't RFC3986 meant for those URI parsers that strictly obey the ABNF? The web-oriented parsers may be excessively liberal in what they accept (Chrome even accepts http://[::1%5D/ and http://[::%31]/ as http://[::1]/, as was pointed out by Philip Homburg). But this behaviour is documented nowhere and should be a bug that must be fixed (even WHATWG has explicitly disallowed this, see https://url.spec.whatwg.org/#example-host-parsing). So I don't think this will make any difference in our final choice of syntax. I would rather say that practicality is the only decisive consideration here.

I'd like to know if you still find the text in RFC3986 confusing after my explanation :)

>> But if we don't, then I'd like to suggest several changes in the text:
>>
>> 1. Clarify that the "%" in an IPv6 scoped address is not used as *data* within a URI, but is instead acting as a subcomponent *delimiter* within the address. This way, the text in the I-D wouldn't somehow conflict with that in RFC 3986.
>
> I would rather say that the words in RFC 3986 are unclear, but that percent-encoding does not apply to this use of % as a delimiter. That's what our new ABNF says.

I believe the only correct way to proceed is we recognise that RFC3986 is clear and self-consistent, or we would only make the words in the draft unclear. I would like to quote the text in Section 3 again and clarify my view on it:

> According to the text in Section 2.4 of [RFC3986], "%" must
> be percent-encoded as "%25" to be used as data within a URI.
> However, in the formal ABNF syntax of RFC 3986, this only applies
> where the "pct-encoded" element appears.  For this reason, it is
> possible to extend the ABNF such that the scoped address
> fe80::abcd%en1 would appear in a URI as http://[fe80::abcd%en1] or
> https://[fe80::abcd%en1].

This part reads as if the text in RFC3986 is not consistent with the formal ABNF syntax of RFC3986, which is quite undesirable. We should try to rephrase it in a well-reasoned manner without repeating the same mistake back in RFC6874.

Thus, I propose that we replace the second paragraph in Section 3 with two paragraphs:

    In a URI, a literal IPv6 address is always embedded between "[" and
    "]".  This document specifies how a <zone_id> can be appended to the
    address. The URI syntax defined by [RFC3986] does not allow the
    presence of a percent ("%") character within an IP literal. For this
    reason, it is backward compatible to allow the use of "%" within an
    IP literal as a delimiter only, such that the scoped address
    fe80::abcd%en1 would appear in a URI as http://[fe80::abcd%en1] or
    https://[fe80::abcd%en1].

    This use of "%" as a delimiter applies only within an IP literal, and
    is irrelevant to and exempt from the percent-encoding mechanism
    [RFC3986].

Since we would have clarified the use of "%" within an IP literal and its exemption from percent-encoding, these statements could be redundant and I propose that we delete them from the text:

    Also, %25 would be valid.

    The <zone_id> "25" cannot be forbidden since it is valid in some
    operating systems, so a parser MUST NOT apply percent decoding to a
    URI such as http://[fe80::abcd%25].

>> 2. If (1) was accepted, add "%" as a reserved character in the "gen-delims" ABNF rule. This makes the I-D consistent with RFC 3986:
>
> I suspect that this would lead to a lot of discussion about unintended side-effects. Since "%" is primarily an escape character, adding it formally as a delimiter would be dramatic. When is it a delimiter and when is it an escape?

That's fair. We might not need to add it formally in the ABNF rule, but I believe we should at least recognise it as a delimiter within an IP literal, just as we recognise "@", ":" and others. The text proposed above should serve well to disambiguate the two uses of "%" within a URI.

>> 3. If (2) was accepted, clarify that "%" is considered as a reserved character only when used within an IPv6 scoped address.
>
> That is surely the effect of the proposed ABNF? Adding it as a comment seems reasonable, though.

See above.

> To be frank, I don't think RFC3986 matters much today, although from an IETF view we need to formally update it. What matters is getting the above "host-representation" text from WHATWG updated.

I mostly agree. Yet it's also worth mentioning that RFC3986 offers consistency in URI parsing and producing for non-browsers.

By the way, I've been writing my own RFC3986-compliant parser and I find it very easy to implement this syntax extension now in RFC6874bis (you'll only need to add a function with 10 lines). So added complexity shouldn't be a concern here. I'll be happy to see this draft improve and advance to the standards track.

Regards,
Shang

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 5 Apr 2022 11:12:29 +1200

Hi Shang,

Thanks very much. I think we will use your new paragraphs, which are indeed much clearer. Personally I think we still need to be explicit that %25 is a valid zone, because I am afraid that not all implementers will understand this (especially since it is unlikely to occur on Linux). I hope there will be a new version in a few days.

Regards
    Brian Carpenter

---

From: "Shang Ye" <[email-masked]>
Date: Tue, 5 Apr 2022 11:00:27 +0800

Brian,

In line:

> Hi Shang,
>
> Thanks very much. I think we will use your new paragraphs, which are indeed much clearer.

Also thanks a lot for taking my advice :)

> Personally I think we still need to be explicit that %25 is a valid zone, because I am afraid that not all implementers will understand this (especially since it is unlikely to occur on Linux). I hope there will be a new version in a few days.

Agree, but I think this statement is still unclear:

> The <zone_id>
> "25" cannot be forbidden since it is valid in some operating systems,
> so a parser MUST NOT apply percent decoding to a URI such as
> http://[fe80::abcd%25].

The problem may be that it doesn't constrain which part of URI a parser must not apply percent-decoding to, and that the "such as" part is a bit ambiguous. For example, should we apply percent-decoding to <http://[fe80::abcd%25]/%25>?

The logic behind this statement also seems unclear. If a parser applied percent-decoding to "fe80::abcd%25" within <http://[fe80::abcd%25]>, would it possibly mean that the parser is forbidding the zone_id "25"? I think it is more likely that the parser is incorrect.

We could be explicit that %25 is a valid zone, but this should probably not the reason why we forbid percent-decoding to an IP literal. Maybe we should split this statement into two halves, put one half immediately after the second paragraph I proposed

    This use of "%" as a delimiter applies only within an IP literal, and
    is irrelevant to and exempt from the percent-encoding mechanism
    [RFC3986]. Implementations MUST NOT percent-decode an IP literal
    within a URI, such as accepting http://[::%31]/ as http://[::1]/.

and the other half elsewhere.

Regards,
Shang

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 6 Apr 2022 11:42:23 +1200

Thanks again, Shang. We will try to ensure that the point is clear in
the next version, especially considering that most parsers don't seem
to refer directly to the ABNF.

Regards
    Brian