Subject: Re: [Rfc-markdown] Carsten: Re: [rfc-i] summary of Removing postal information from RFCs

From: Carsten Bormann <[email-masked]>
Date: Mon, 19 Jul 2021 20:34:17 +0200

CCing RFC-markdown...

> On 19. Jul 2021, at 19:18, Toerless Eckert <[email-masked]> wrote:
> 
> So, what is the relationship kramdown-rfc2629 to this ?
> 
> - Can the kramdown create xmlv3 output ?

Yes, via the --v2v3 process in xml2rfc.

Without the --v3 (kdrfc -3) flag, kramdown-rfc2629 produces just that, RFC 2629 (or its evolution that we call v2, as per RFC 7749).
This can be used by the historic v2 code in xml2rfc, or it can be converted to v3 using xml2rfc —v2v3.
After the successful transition from v2 to some v3, I don’t think that is a very function any more, but I’ll wait before killing it.

With the --v3 (-3) flag, kramdown-rfc2629 produces XML that xml2rfc can process, and that --v2v3 can turn into the current version of v3.
This XML is no longer v2 compliant, as it uses the new features of v3.
Not all v3 features are directly accessible from markdown syntax, but practically all of them can be activated via inline attribute lists (IALs) or by including XML syntax in the markdown.
(The only one really missing from the direct markdown translation is the new table format with colspan/rowspan, and that is hard to get right in markdown — I’m working on it.
Until then just enter the tables that need this in v3 XML format inside the markdown.)

> - Does it support postalline ?

Since 1.5.2, yes.
(The memo: update early, update often.)

> - Is there a better way to figure out answers other han
>   - check readme on github (didn't find it)

I’d memoize this: http://rfc.space
(Yeah, I’ll get to httpsing this, one of these days.)

>   - ask carsten (here)

Here = [email-masked] (which can also be searched).

>   - trial and error (tried to avoid this so far ..)

Yes: read the commit history and/or the source.
Important progress is announced on [email-masked], but since postalLine support is probably a minority program right now, I didn’t announce 1.5.2 there yet (too little time before today).

Re postalLine, the 1.5.2 commit says:

+  # The below anticipates the "postalLine" changes.
+  # If a postalLine is used (abbreviated "postal" in YAML),
+  # non-postalLine elements are appended as further postalLines.
+  # This prepares for how "country" is expected to be handled
+  # specially with the next schema update.
+  # So an address is now best keyboarded as:
+  #   postal:
+  #     - Foo Street
+  #     - 28359 Bar
+  #   country: Germany

So what happens here is to put the value of `country:` in as another postalLine until `<country>` becomes allowed together with `<postalLine>`s in one of the next v3 versions.
Then there will be another change in the tool (here my strategy to isolate kramdown-rfc from v3 changes via --v2v3 obviously doesn’t work).
If you follow the above recommendation, there will be no need to do another change to the YAML then.

Grüße, Carsten

> 
> Cheers
>    Toerless
> 
> On Wed, Jul 14, 2021 at 09:29:12AM +0200, Carsten Bormann wrote:
>> On 14. Jul 2021, at 03:57, John R Levine <[email-masked]> wrote:
>>> 
>>> Or you can give a sequence of postallines with an optional country, in which we render the whole list in order.
>> 
>> Works for me.
>> 
>> (I’m sure nobody will mind if my postal people tell me that starting the address with the lines
>> 
>> GND: 987654321
>> ORCID: 0123-4711-0815
>> 
>> will get my mail faster to me.)
>> 
>> Grüße, Carsten
>> 
>> _______________________________________________
>> rfc-interest mailing list
>> [email-masked]
>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
> 
> -- 
> ---
> [email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 19 Jul 2021 21:08:11 +0200

> With the --v3 (-3) flag, kramdown-rfc2629 produces XML that xml2rfc can process, and that --v2v3 can turn into the current version of v3.

… and if you want to submit this as an I-D, use

	kdrfc -3c foo.md

and submit the foo.v2v3.xml *ONLY*.

(I’ll probably add API submission soon.)

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Tue, 20 Jul 2021 09:28:07 -0400

> Yes, via the --v2v3 process in xml2rfc.

Do you think that you'll change to producing pure v3 output at some point?
I understand that might be awhile, or that -3 might become the default before
that point.

I'm just asking about processing plans.

    > With the --v3 (-3) flag, kramdown-rfc2629 produces XML that xml2rfc can
    > process, and that --v2v3 can turn into the current version of v3.

So it's some kind of v2 / v3 mutt :-)

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2021 15:59:25 +0200

> Do you think that you'll change to producing pure v3 output at some point?

Yes.

But since most processing of RFCXML involves xml2rfc already, and that already has a (by definition) perfect v3 generator, this doesn’t have very high priority.
Any mistake I make there in kramdown-rfc leads to a show-stopper for an author, and that is exactly what kramdown-rfc is trying never to create.

> I understand that might be awhile, or that -3 might become the default before

Right.  Again, there is very little reason to invalidate existing Makefiles etc., so changing defaults pretty much means making a renamed copy of a wrap-around processor like kdrfc.
That is actually likely to happen quite soon.

(-3 is already default in, and the only way to run, kdwatch [1].)

> I'm just asking about processing plans.

I think progress here will be pretty much issue-driven; if an author finds an inconvenience in the current setup, that is more likely to be addressed than some abstract notion of compliance.
(The main driver is actually AUTH48 processing, as the RPC operates on the converted v3 XML, so making diffs is harder for the authors when there are still pieces of v2 packaging around.  But making that process less impenetrable for authors is currently frustrated in more ways, so that is a larger project.  The other driver is missing features, mainly tables at this point.)

> So it's some kind of v2 / v3 mutt :-)

It is pretty much one of the two authoring interfaces provided by xml2rfc to v3; it is just weird that the I-D submission interface is in denial about that.

Grüße, Carsten

[1]: https://github.com/cabo/kdwatch

---

From: Miek Gieben <[email-masked]>
Date: Tue, 20 Jul 2021 16:27:18 +0200

[ Quoting <[email-masked]> in "Re: [Rfc-markdown] Carsten: Re: [rf..." ]
>Yes, via the --v2v3 process in xml2rfc.

FWIW mmark opted to just output v3 XML directly (I've deleted the whole XML2 rendering),
but this does require to be up to date with xml2rfc.

Postalline is supported.

/Miek

--
Miek Gieben

---

From: Jay Daley <[email-masked]>
Date: Wed, 21 Jul 2021 08:55:09 +1200

> On 21/07/2021, at 1:59 AM, Carsten Bormann <[email-masked]> wrote:
> 

I’m confused.  How is this different from the pre-v3 situation where kramdown-rfc was producing v2 and that could be rejected by xml2rfc if there was a mistake in it?

Jay

> Rfc-markdown mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfc-markdown

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Jul 2021 23:36:58 +0200

Hi Jay,

> I’m confused.  How is this different from the pre-v3 situation where kramdown-rfc was producing v2 and that could be rejected by xml2rfc if there was a mistake in it?

Beautiful question.

If this were steady-state situation, I could only point out that xml2rfcv1 was extremely permissive and xml2rfcv2 tried to accommodate as many of these idiosyncrasies as possible.  So writing for xml2rfcv2 was much easier to get right than for xml2rfcv3.

But that is the less interesting part of the answer.  The more interesting one is that kramdown-rfc was written at a time when RFCXML was pretty much stable.  Kramdown-rfc had a few years with a small number of users to get the bugs sorted out, including the bugs that only matter to the users.  Documents that can be processed with the 2012 version still process with the 2021 version.  That is an important characteristic that I think should be (and is!) maintained across the v2v3 transition.
Much of that transition is busywork — rewriting stuff to get the same functionality we already had.  Just with new risks.
Since xml2rfc already has a conversion mechanism that works incredibly well, it would have been less than bright not to use that mechanism instead of redoing all that work in kramdown-rfc.

Couple that with weird stuff like ascii attributes (23 of them) popping up on all kinds of elements with text content, or the rather broken content model of <li>, the <dl> regression etc. — this increases the busywork and also makes it more likely that v3 will again and again change in subtle ways to fix these rough patches.
It’s not that I didn’t start attempts to do wholesale conversions, but each time I did, I ran into one of these subtle things.  These have (mostly) been solved in --v2v3, so why not use that?

Much of the parts of v3 that aren’t just gratuitously different (or, “cleaned up”, depending on perspective) but provide new functions were easy to integrate.
Who ever uses these knows they will be get shot full of arrows (think “referencegroup”, or “svg”), so we can do some experimentation.

My objective with writing kramdown-rfc was to free myself from XML.
My objective with publishing it was to allow others to free themselves from XML.
But in both cases, we need to take co-authors with us that may not all be in “now I hate it every time I need to use XML again” territory yet.
Actually, many long-time RFC writers have serious Stockholm syndrome, and more than one collaboration that I tried to start didn’t work because of that.
So it is even more important not to create show-stoppers, because a single one of those can cause hostages to pack up and go back into the bank.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Wed, 21 Jul 2021 19:34:22 +0200

Am 20.07.2021 um 23:36 schrieb Carsten Bormann:
> ...
> ...

Could you elaborate on "the rather broken content model of <li>"?

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 22 Jul 2021 02:53:50 +0200

On 21. Jul 2021, at 19:34, Julian Reschke <[email-masked]> wrote:
> 
> Could you elaborate on "the rather broken content model of <li>”?

Sorry, it’s been a while.
I started converting kramdown-rfc’s ol/ul to v3 and ran into an issue, which made me stick with v2 syntax, letting xml2rfc do the work.
Don’t remember what it was.  I seem to remember there is still one bug in xml2rfc here, as well.
(These things pop up when I’m trying to do something else…
I don’t always take good notes when I’m at yak shaving stack depth 5 already.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 26 Jul 2021 11:03:53 +0200

Public service announcement:

The, er, characteristic of the submission process explained below has been fixed during the pre-111 moratorium.
So, starting from yesterday, you can now directly submit the kramdown-rfc generated hybrid v3 XML as Internet-Drafts; no additional step via -3c is needed any more.
(The improvement of course also lets non-markdown drafts benefit that are edited directly in RFCXMLv2 or hybrid v3 XML.)

Thanks to the tools team!

Grüße, Carsten

> (I’ll probably add API submission soon.)

[1]: <https://mailarchive.ietf.org/arch/msg/wgchairs/EikOARXSYfKz9-D-YLSjNaVN7MU>

 * Merged in [19242] from [email-masked]:
   Run v2v3 converter internally when v2 XML is submitted. Fixes #3305.  

(My more detailed description of the problem in
<https://mailarchive.ietf.org/arch/msg/ietf-and-github/-neyBKvA24nCKh1r2ELz4taGz7s>
contains a recommendation to do the v2v3 conversion locally before submission — this is of course still possible, but unless the I-D relies on a very recent xml2rfc fix that you have installed locally but may not have found its way into datatracker, no longer necessary.)

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 27 Jul 2021 09:58:20 +1000

Thanks, Carsten! I've converted one of my drafts' toolchain to use this, and it seems to work well (with some differences in the XML, which is to be expected).

One question: do you have any plans to tidy the XML output (e.g., with indentation, etc.)?

Cheers,

> On 26 Jul 2021, at 7:03 pm, Carsten Bormann <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Jul 2021 02:13:49 +0200

> Thanks, Carsten! I've converted one of my drafts' toolchain to use this, and it seems to work well (with some differences in the XML, which is to be expected).

The XML left in the drafts repository by the current submission engine is the unconverted one.

> One question: do you have any plans to tidy the XML output (e.g., with indentation, etc.)?

There actually is code to do some indentation.

There have never been bug reports on this, so it seems it is mostly non-functional now.

What are the requirements on that tidying?

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 27 Jul 2021 10:17:24 +1000

> On 27 Jul 2021, at 10:13 am, Carsten Bormann <[email-masked]> wrote:
> 

Sorry, I don't understand?

> What are the requirements on that tidying?

Just aesthetic; if the XML is going to be the canonical format, it's nice if it's somewhat pretty. Also, it reduces the chances of the RPC making unnecessary diffs in their work.

Cheers,

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Jul 2021 02:22:34 +0200

> Sorry, I don't understand?

The XML when you don’t convert before sending is the one that gets to the I-D repo, so it certainly will look different.

> Just aesthetic; if the XML is going to be the canonical format, it's nice if it's somewhat pretty.

But the RPC makes sure of that :-)

> Also, it reduces the chances of the RPC making unnecessary diffs in their work.

They will start from the v2v3 output, so there will be tons of diffs from an unconverted hybrid document.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 27 Jul 2021 10:28:18 +1000

> On 27 Jul 2021, at 10:22 am, Carsten Bormann <[email-masked]> wrote:
> 

Sorry, now I'm really confused.

My understanding was that (with the appropriate flags), kramdown-rfc2629 will now generate v3 output. That's what I've changed my toolchain to do. Are you saying that the submission tool will somehow munge that back to v2, then up to v3 again?

Cheers,

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Jul 2021 02:32:48 +0200

> 
> My understanding was that (with the appropriate flags), kramdown-rfc2629 will now generate v3 output.

No, it generates hybrid output (v3 elements in a v2 shell).
(The real reason is that if you put them in a v3 shell, xml2rfc suddenly becomes much more autocratic.)

> That's what I've changed my toolchain to do. Are you saying that the submission tool will somehow munge that back to v2, then up to v3 again?

The submission tool will use xml2rfc to process the hybrid document, which provides all the benefits of v3, but is more lenient.

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 27 Jul 2021 10:34:25 +1000

I am left with the feeling that v3 is still not really ready for production use.

> On 27 Jul 2021, at 10:32 am, Carsten Bormann <[email-masked]> wrote:
> 
>>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Jul 2021 03:31:01 +0200

> I am left with the feeling that v3 is still not really ready for production use.

Well, there is no way around the fact that the RPC is using it for its P.

So we somehow need to make it work for ourselves.

And there is good stuff in there, much better tables, relative references, support for a contributor section that looks like an author section, SVG support, text (and not just text string) support for headings (except that some of it currently seems to break ToC generation), I’m forgetting half of it because I already take it for granted.

You will never get the feedback you need to make these actually usable without pressing them into production.

And the transition strategy to provide a permissive hybrid as the input language for XML2RFC, with an option to generate a restricted, “beautiful” canonical form from that, is much better in practice than theory would make one believe.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 27 Jul 2021 09:40:02 +0200

> I am left with the feeling that v3 is still not really ready for production use.

I won't repeat my opinion about the recent v3 process here :-)

But with that said: if there's a bug, raise an issue. If there's a
feature missing: raise an issue.

Just stop generating v2 :-)

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Jul 2021 12:00:57 +0200

> Just stop generating v2 :-)

There was a smiley here, but some people might take this at face value.

As long as

     <?rfc include="reference.RFC.2119”?>

no longer works in v3, I’ll recommend people who can’t (*) convert to markdown to stay with v2 (preferably processed in hybrid mode).

This one specimen is, of course, just an example of how v2 was designed a little bit more for authors, while v3 is prioritizing a form of standards compliance that actually is not giving us anything on the author side.

How is the below better than the above:

    <xi:include href="https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>

The v3 version encodes a URI that is brittle (you would probably already tell me that the authority component is wrong now, and the path name also is brittle).  It requires xi:include processing in any additional author-side tools.  And it is impossible to type from memory.

I’m a German, so I do have the genes for excessive bureaucracy.
But this…  No.

Grüße, Carsten

(*) You co-author might retire soon and this is “their last RFC”, so they don’t want to learn something new while they have ingrained RFCXMLv2 skill.  Or some such.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Jul 2021 12:41:42 +0200

> no longer works in v3, I’ll recommend people who can’t (*) convert to markdown to stay with v2 (preferably processed in hybrid mode).

Note I said "stop generating", as opposed to "stop authoring".

> The v3 version encodes a URI that is brittle (you would probably already tell me that the authority component is wrong now, and the path name also is brittle).  It requires xi:include processing in…

The fact that the URI is brittle is entirely the fault of the IETF. This
should have been fixed years ago. Cool URIs do not change.

My experience is that people copy & paste from existing documents or
references, so I really don't buy the "type from memory" argument.

The issue with the v2 PI is that is was never specified, and has
heuristics for document types wired in. Plus local caching. There's a
reason why a huge number of XML processed in AUTH48 before the switch
now does not compile anymore.

Yes, one could argue that for the common cases, a simpler syntax would
be nice. -> Feature request.

> (*) You co-author might retire soon and this is “their last RFC”, so they don’t want to learn something new while they have ingrained RFCXMLv2 skill.  Or some such.

Then run through the converter, and edit from there?

Best regards, Julian