Subject: [rfc-i] looking for a volunteer to write a simple script

From: Heather Flanagan <[email-masked]>
Date: Fri, 12 Jul 2019 09:55:04 -0700

Hola a todos!

The RFC Editor has the need for a comparatively simple script that would automatically add <bcp14></bcp14> tags to requirement language in v3 RFCs.

Specifically, this would take a v3 XML input file, and create a v3 XML output file with <bcp14></bcp14> added around each instance of a 2119 keyword in the file. (MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL)

Anyone up for helping us out with that?

Thanks! Heather
_______________________________________________
rfc-interest mailing list
[email-masked]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest

---

From: Russ Housley <[email-masked]>
Date: Fri, 12 Jul 2019 13:12:51 -0400

Heather:

I assume this needs to be a command line tool.  Do you care about the language used?  bask script, python script, whatever?

Russ

> On Jul 12, 2019, at 12:55 PM, Heather Flanagan <[email-masked]> wrote:
> 
> Hola a todos!
>

---

From: Heather Flanagan <[email-masked]>
Date: Fri, 12 Jul 2019 10:14:12 -0700

Yes to the command line, no strong preference re: language beyond “please don’t use something arcane that will require unusual libraries be installed on the RFC Editor server”.

Thanks! Heather

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 10:14:17 -0700

This is fairly simple; the only question is what OS and/or language. 

A simple solution would be to use Perl and run as a command line. 

I can knock this out this evening if you like. It would help to have a
few .xml example files to test (or a web pointer). 

Some quick questions, though: 

- do we need to check for hyphenated variants? 

- do we need/want to check for lower-case uses and issue warnings if
found (to the error output, not in the modified file) 

Joe

---

From: Julian Reschke <[email-masked]>
Date: Fri, 12 Jul 2019 19:23:08 +0200

On 12.07.2019 18:55, Heather Flanagan wrote:
> Hola a todos!
>
> The RFC Editor has the need for a comparatively simple script that would
> automatically add <bcp14></bcp14> tags to requirement language in v3 RFCs.
>
> Specifically, this would take a v3 XML input file, and create a v3 XML
> output file with <bcp14></bcp14> added around each instance of a 2119
> keyword in the file. (MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT,
> SHOULD, SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL)
>
> ...

The tricky part is to find the right instances. For instance, what if it
appears in a quote, or in artwork? Or if "SHALL NOT" is across a line
break...

So the output will require sanity checking.

I assume that the tool is supposed to preserve whitespace, line breaks
etc? This essentially rules out running the input through an XML parser...

Best regards, Julian

---

From: Heather Flanagan <[email-masked]>
Date: Fri, 12 Jul 2019 10:24:45 -0700

> On Jul 12, 2019, at 10:14 AM, Joe Touch <[email-masked]> wrote:
> 

OS = Linux
> I can knock this out this evening if you like. It would help to have a few .xml example files to test (or a web pointer).
> 

No.

> - do we need/want to check for lower-case uses and issue warnings if found (to the error output, not in the modified file)
> 
> 

No. Too many false positives.

Thanks!

>> [email-masked] <mailto:[email-masked]>
>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>_______________________________________________

---

From: Heather Flanagan <[email-masked]>
Date: Fri, 12 Jul 2019 10:26:26 -0700

> On Jul 12, 2019, at 10:23 AM, Julian Reschke <[email-masked]> wrote:
> 

Well, yes, of course. We’re aiming for a rough pass to catch maybe 80% of the situations. Everything will still need to be reviewed.

> etc? This essentially rules out running the input through an XML parser…

Seriously, we’re not aiming for that robust right now. It doesn’t have to be perfect, it just has to help.

-Heather

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 10:30:52 -0700

> appears in a quote, or in artwork? Or if "SHALL NOT" is across a line

That's actually quite easy in Perl - including retaining the original
line breaks/whitespace. 

> etc? This essentially rules out running the input through an XML parser...

That shouldn't be necessary. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 10:43:32 -0700

This will do the trick: 

perl -0777 -pe
"s/(((MUST|SHOULD|SHALL)(\s+NOT)?)|((NOT\s+)?RECOMMENDED)|MAY|OPTIONAL|REQUIRED)/<bcp14>\$1<\/bcp14>/g"
INFILE.xml > OUTFILE.xml 

(replace INFILE.xml and OUTFILE.xml with your filenames) 

If you want it to edit in-place (riskly, but simpler if you work on a
copy anyway): 

perl -0777 -i -pe
"s/(((MUST|SHOULD|SHALL)(\s+NOT)?)|((NOT\s+)?RECOMMENDED)|MAY|OPTIONAL|REQUIRED)/<bcp14>\$1<\/bcp14>/g"
INFILE.xml > OUTFILE.xml

> So the output will require sanity checking.

Well, yes, of course. We're aiming for a rough pass to catch maybe 80%
of the situations. Everything will still need to be reviewed. 

> etc? This essentially rules out running the input through an XML parser...

Seriously, we're not aiming for that robust right now. It doesn't have
to be perfect, it just has to help. 

-Heather

---

From: Julian Reschke <[email-masked]>
Date: Fri, 12 Jul 2019 19:49:07 +0200

> That shouldn't be necessary.

I assume that will run before AUTH48.

It would be *very* unfriendly to the document editors to generate a huge
diff just because a tool felt like reformatting everything.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 12 Jul 2019 19:52:47 +0200

I would spend a couple of \b (or \< and \>, I don't remember Perl well enough), so MARSHALL in MAYNARD isn't hit. (Theresa is hopeless, but we knew that.)

Sent from mobile

> On 12. Jul 2019, at 19:43, Joe Touch <[email-masked]> wrote:
> 
> This will do the trick:
> 
> 
> 
> perl -0777 -pe "s/(((MUST|SHOULD|SHALL)(\s+NOT)?)|((NOT\s+)?RECOMMENDED)|MAY|OPTIONAL|REQUIRED)/<bcp14>\$1<\/bcp14>/g" INFILE.xml > OUTFILE.xml
> 
> If you want it to edit in-place (riskly, but simpler if you work on a copy anyway):
> 
> perl -0777 -i -pe "s/(((MUST|SHOULD|SHALL)(\s+NOT)?)|((NOT\s+)?RECOMMENDED)|MAY|OPTIONAL|REQUIRED)/<bcp14>\$1<\/bcp14>/g" INFILE.xml > OUTFILE.xml
> 
>  
> 
> 
>> Well, yes, of course. We're aiming for a rough pass to catch maybe 80% of the situations. Everything will still need to be reviewed.
>> 
>>> 
>> Seriously, we're not aiming for that robust right now. It doesn't have to be perfect, it just has to help.
>>  
>> -Heather
>>  
>> 
>>

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 11:02:58 -0700

Quick update: 

perl -0777 -pe
"s/(\b(((MUST|SHOULD|SHALL)(\s+NOT)?)|((NOT\s+)?RECOMMENDED)|MAY|OPTIONAL|REQUIRED)\b)/<bcp14>\$1<\/bcp14>/g"
INFILE.xml > OUTFILE.xml 

Joe

-Heather

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 11:07:58 -0700

> On Jul 12, 2019, at 10:52 AM, Carsten Bormann <[email-masked]> wrote:
> 

Ah yes - fixed.

---

From: Sandy Ginoza <[email-masked]>
Date: Fri, 12 Jul 2019 11:29:59 -0700

Thanks Joe!

I just tested this updated script and it seems to work well.  I am happy to see it catches things like “MAY” (within quotes), but does not tag items Carsten noted (e.g., “MARSHALL”).   

I note that it does double tag keywords if they’ve already been tagged, but that should be something we can check for before running the script (i.e., check whether the keywords have already been tagged).  

Thanks!
Sandy

> On Jul 12, 2019, at 11:02 AM, Joe Touch <[email-masked]> wrote:
> 
> Quick update:
> 
> perl -0777 -pe "s/(\b(((MUST|SHOULD|SHALL)(\s+NOT)?)|((NOT\s+)?RECOMMENDED)|MAY|OPTIONAL|REQUIRED)\b)/<bcp14>\$1<\/bcp14>/g" INFILE.xml > OUTFILE.xml
> 
> 
> 
> Joe
> 
>  
>> On Jul 12, 2019, at 10:23 AM, Julian Reschke <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 11:33:12 -0700

I can fix that in a few minutes...

> On Jul 12, 2019, at 11:29 AM, Sandy Ginoza <[email-masked]> wrote:
> 
> Thanks Joe!
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 17:16:39 -0700

> On Jul 12, 2019, at 10:49 AM, Julian Reschke <[email-masked]> wrote:
> 

I was indicating that we don’t need to use an XML parser.

It’s trivial to retain formatting to minimize diff output. The code I gave does that already.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Jul 2019 17:19:27 -0700

I looked at this a bit and getting it closer to 100% is tricky. There are a LOT of corner cases, including the possibility that the <bcp14> tags are outside other tags, e.g., bold, etc.

I don’t know fi it’ll be worth the effort, given the need for post-checking anyway.

But I’ll take a look over the weekend and let you know if there’s a solution that gets closer without increasing falses.

Joe

> On Jul 12, 2019, at 11:33 AM, Joe Touch <[email-masked]> wrote:
> 
> On Jul 12, 2019, at 11:29 AM, Sandy Ginoza <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> Thanks Joe!
>> 
>>> On Jul 12, 2019, at 11:02 AM, Joe Touch <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> Quick update:
>>>

---

From: Dick Franks <[email-masked]>
Date: Sat, 13 Jul 2019 13:23:47 +0100

WARNING: banning rules detected suspect part(s),
do not open unless you know what you are doing

Subject: Re: [rfc-i] looking for a volunteer to write a simple script
From: Dick Franks <[email-masked]>
To: Heather Flanagan <[email-masked]>
Cc: RFC Interest <[email-masked]>
Date: Sat, 13 Jul 2019 13:23:47 +0100

More robust solution that recognises keywords within quotes and/or
split across line break.

Dick Franks
________________________

---

From: Heather Flanagan <[email-masked]>
Date: Sun, 14 Jul 2019 09:49:49 -0700

> On Jul 12, 2019, at 5:19 PM, Joe Touch <[email-masked]> wrote:
> 

The IETF absolutely *lives* for corner cases. :-) We are expecting those, and it’s ok. There is no expectation of 100% here.

> But I’ll take a look over the weekend and let you know if there’s a solution that gets closer without increasing falses.

You’re a gem! Thanks!

Heather

> 
> Joe
> 
>> On Jul 12, 2019, at 11:33 AM, Joe Touch <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Joe Touch <[email-masked]>
Date: Sun, 14 Jul 2019 21:55:13 -0700

Hi, Heather,

There are two solutions, attached. They both avoid duplicate insertion of the tags, but ONLY if the tags were used adjacent to the keywords with only whitespace between.

simple-bcp-fix.pl
	runs on any version of perl5
	does a simple search and replace that may create duplicates if <bcp14> tags are already in use
	but then removes duplicates *if they are separated only by whitespace*

lookback-bcp-fix.pl
	runs only on perl 5.30.0 and later, using an experimental feature “variable lookback"
	inserts tags if the tag isn’t immediately preceding, separated by up to 248 chars of whitespace
	no need to remove duplicates

you can test which version of perl you’re running using “perl -v”. if it’s 5.30.0 or later, either version will run. If lower than 5.30.0, then only the first (simple-bop-fix) will work.

to run, use it this way (e.g., for using simple-bcp-fix.pl):

	perl simple-bcp-fix.pl INFILE.xml > OUTFILE.xml

to modify files in-place, use:

	perl -i simple-bcp-fix.pl INFILE.xml

Joe

---

From: Dick Franks <[email-masked]>
Date: Mon, 15 Jul 2019 11:04:22 +0100

Subject: Fwd: [rfc-i] looking for a volunteer to write a simple script
From: Dick Franks <[email-masked]>
To: RFC Interest <[email-masked]>
Date: Mon, 15 Jul 2019 11:04:22 +0100

---------- Forwarded message ---------
From: Dick Franks <[email-masked]>
Date: Sun, 14 Jul 2019 at 20:24
Subject: Re: [rfc-i] looking for a volunteer to write a simple script
To: Russ Housley <[email-masked]>
Cc: Heather Flanagan <[email-masked]>, Robert Sparks
<[email-masked]>, Henrik Levkowetz <[email-masked]>

Russ,

Happy to do that.
Revised version attached.

Dick
________________________

On Sun, 14 Jul 2019 at 18:15, Russ Housley <[email-masked]> wrote:
>
> Dick:
>
> Any chance that you would be willing to contribute this under the Simplified BSD License?
>
> Please see pages 3 and 4 of https://trustee.ietf.org/license-info/IETF-TLP-5.pdf
>
> Russ
>