Subject: [Cbor] Regular expressions

From: Joe Hildebrand <[email-masked]>
Date: Sun, 28 Feb 2021 12:03:01 -0700

I know that rfc8949 dropped Regular Expressions as tag 35, but they're still defined and usable from rfc7049.

That said, what, if anything, are people doing the the regex flags?  I wonder if all this time I should have been encoding

/foo/g

As 

35("/foo/g")

Instead of 

35("foo")

(Losing the g flag).  I see from https://mailarchive.ietf.org/arch/msg/cbor/txIKHMXRFzNo7oH-eHZigO1L47w/ that Carsten has previously also assumed that we don't transport the slashes.

If including the slashes won't interop, and if I'm not the only one that's implemented tag 35, I'd be happy to whip up a new doc to describe this, and register a higher tag number for it.  If anyone is interested, we can have a discussion about

35("/foo/g") vs. 35("foo", "g")

And maybe even some sort of info about what kind of regex it is (ECMAscript vs. PCRE, for example).

I assume a lot of folks are in the "regexes are too hard to interop" camp, in which case I'll take a nice high tag number and everyone else can ignore it.

— 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 28 Feb 2021 21:29:07 +0100

> I know that rfc8949 dropped Regular Expressions as tag 35, but they're still defined and usable from rfc7049.

Right.

> That said, what, if anything, are people doing the the regex flags?  I wonder if all this time I should have been encoding

The intention certainly was to give the RE, not a RE literal.
So the slashes and the g would become part of the RE, and

35(“/foo/g”) = RegExp(“/foo/g”) = /\/foo\/g/

> (Losing the g flag).  I see from https://mailarchive.ietf.org/arch/msg/cbor/txIKHMXRFzNo7oH-eHZigO1L47w/ that Carsten has previously also assumed that we don't transport the slashes.

Right.

> 35("/foo/g") vs. 35("foo", "g")

I can’t speak about the g flag (it is not actually an RE modifier), but, e.g. for the literal

/foo/i 

its value in PCRE is 

(?i)foo
or
(?i:foo)

so there is no need to carry the modifiers as flags outside the RE.

> And maybe even some sort of info about what kind of regex it is (ECMAscript vs. PCRE, for example).

Definitely.  And for non-PCRE, there may be a need to carry flags outside.
ECMAscript (JavaScript) and PCRE are quite close, but there are other families as well:
RFC 8610 uses W3C Schema (WSD)-types regexes, as does YANG (at least officially, not so much in practice).  These are anchored (good) and support subtraction (exceedingly good), but are stuck with Unicode complexity in \d and \w, even in \s, which are therefore not useful in ASCII protocols.

> I assume a lot of folks are in the "regexes are too hard to interop" camp, in which case I'll take a nice high tag number and everyone else can ignore it.

They are hard, but not “too hard” for a large number of applications.
Enjoy https://www.regular-expressions.info/refmodifiers.html
just for the modifiers :-)
Flags such as g in JavaScript or u and n in Ruby are much harder to express inside the RE.

(But I still prefer ABNF :-)

Grüße, Carsten

---

From: Joe Hildebrand <[email-masked]>
Date: Sun, 28 Feb 2021 13:38:23 -0700

> I can’t speak about the g flag (it is not actually an RE modifier), but, e.g. for the literal

In ECMAscript land 'gimsuy' are all valid.

> so there is no need to carry the modifiers as flags outside the RE.

Hm.  I don't think I can use that construct, but I'm happy to learn yet another thing about regex incompatibility. :)

> RFC 8610 uses W3C Schema (WSD)-types regexes, as does YANG (at least officially, not so much in practice).  These are anchored (good) and support subtraction (exceedingly good), but are stuck with…

Nod.  We'd probably need a small registry then, with the names or a code.  I would expect the semantics are "use this if it's a type you know about, otherwise, keep the tagged version and punt to the application layer".

> just for the modifiers :-)

zow.

> (But I still prefer ABNF :-)

No argument, but I use regex's every day, and ABNF or a full PEG grammar just when I need to get out the big hammer.

— 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 00:24:10 +0100

> In ECMAscript land 'gimsuy' are all valid.

But g, for instance, is not an RE modifier.  It modifies how the operations are using it, but not the RE itself.  Similar for y.  u is really enabling syntax for decoding unicode, so it should not be visible during interchange (of the decoded RE).

> Nod.  We'd probably need a small registry then, with the names or a code.  I would expect the semantics are "use this if it's a type you know about, otherwise, keep the tagged version and punt to t…

Right.  But why not use tags for those?  We already have that registry.

(We could write a common document that we expect new RE tag registrations to reference, so there is some structure to this.)

> No argument, but I use regex's every day, and ABNF or a full PEG grammar just when I need to get out the big hammer.

REs are certainly more amenable to interchange as such.

I need to fix up my ABNF to RE compiler…
(Which is almost trivial – as long as the ABNF is not recursive – but the current version generates way too much noise that a manual RE writer would know how to avoid.)

I’m also on the lookout for a toolkit for translating between the various RE dialects.

Grüße, Carsten

---

From: Joe Hildebrand <[email-masked]>
Date: Sun, 28 Feb 2021 16:34:19 -0700

> On Feb 28, 2021, at 4:24 PM, Carsten Bormann <[email-masked]> wrote:
> 

Hm.  I understand your point, but round-tripping the objects intact is more important to me than which part is technically part of the regex.

> (We could write a common document that we expect new RE tag registrations to reference, so there is some structure to this.)

That works for me.  So, there would be an "ECMAscript RegExp" tag then, right?

> I’m also on the lookout for a toolkit for translating between the various RE dialects.

I bet the regex101 folks have a starting point for that.  Does anyone have a contact there?

BTW, one of the reasons all of this has come up for me is I've been tinkering with Basura (https://github.com/hildjj/basura/), a library/CLI for generating random trash JS that is syntactically valid, but otherwise useless.  I just released it this morning, and I've learned a few interesting things about my CBOR implementation by tying the two together.

— 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 00:38:58 +0100

> That works for me.  So, there would be an "ECMAscript RegExp" tag then, right?

Probably more than one, as ECMAscript is evolving heavily.
“ECMAscript RegExp” before 2015 is radically less comprehensive than the 2020 version.

And it would probably be an “ECMAscript RegExp literal”.

Grüße, Carsten

---

From: Joe Hildebrand <[email-masked]>
Date: Sun, 28 Feb 2021 16:40:26 -0700

> On Feb 28, 2021, at 4:38 PM, Carsten Bormann <[email-masked]> wrote:
> 

"ECMAscript 2020 RegExp literal" works for me.  What are our next steps?

— 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 1 Mar 2021 00:45:48 +0100

> "ECMAscript 2020 RegExp literal" works for me.  What are our next steps?

How quickly do you need them?

I think I’d like to write a section about RE tags in 
https://cabo.github.io/notable-tags/draft-bormann-cbor-notable-tags.html

but that doesn’t mean it needs to be the origin of the registrations.
You could also go for an FCFS tag today (in my timezone :-) and add specification text later.

Grüße, Carsten

---

From: Joe Hildebrand <[email-masked]>
Date: Sun, 28 Feb 2021 16:50:03 -0700

> On Feb 28, 2021, at 4:45 PM, Carsten Bormann <[email-masked]> wrote:
> 

No hurry.  I'd rather get it right.

> https://cabo.github.io/notable-tags/draft-bormann-cbor-notable-tags.html

Added to my queue, thanks.

— 
Joe Hildebrand