Subject: Re: [yang-doctors] YANG Tool Requests => regex validator

From: Radek Krejčí <[email-masked]>
Date: Tue, 1 Aug 2017 11:30:49 +0200

Hi,
yangre requires patterns as you would write them in a YANG schema. Using { in pattern without quoting is illegal - see RFC 7950, sec 6.1.3. It is the same issue as have semicolon there. So, yangre actually does not fail with quantifier, but since it follows {, a valid YANG statement is expected, but 3 is an invalid statement/keyword.

yangre actually takes the pattern, creates very small YANG schema and then 1) validates the schema (which does not pass in this case) and then 2) checks if the leaf of string type with the specified pattern(s) can have the provided value (if it matches the patterns). So it actually checks what yanglint does in case the string is specified as leaf's default value.

Regarding the quotes in CLI - it is quite challenging to make quotation correct because of interpreting quotation by shell. See help, there are some examples but there can be some special cases or different behavior with different shell. The current yangre accepts alternative input via a file (patterns on separate lines, leading space for invert match, at the last line after a single blank line the string for matching) which would avoid issues with quoting. Anyway, as we were discussing with Pieter - Python seems to do the quotation correct (or probably just bypass the shell), but yangre expect the patterns _including_ the quotation as you would write it in YANG schema. Only this way all the YANG rules can be checked.

Regards,
Radek

Dne 29.7.2017 v 12:06 Pieter Lewyllie (pilewyll) napsal(a):
> I get the same results as Joe stated before.
>
> I’m wondering why YANGRE requires extra quotes to match with the {3} quantifier.
>
> Using the GUI:
>
> (([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3} matches 192.1.1. on W3CGREP, but not on YANGRE ( yangre error: Invalid keyword "3”. )
>
> '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}’ matches 192.1.1. on YANGRE, but not on W3CGREP (which is expecting the ‘ in the test string)
>
> The same behaviour can be observed in the CLI, you just have to add wrap the expressions with appropriate quotes.
>
> *Radek,* do you have an idea why the YANGRE fails with the quantifier? Here are a couple of CLI examples that I observed:
> yangre -p "(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?" "192.168.0.1"
> yangre error: Invalid keyword "3”.
>
> /yangre -p "'(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?'" “192.168.0.1” 
> matches (no output)
>
> When inverting the quotes: (single quotes on the outside)
> yangre -p '"(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?"' "192.168.0.1"
> yangre error: Invalid keyword "(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?”".
>
> yangre -p '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?' "192.168.0.1"
> yangre error: Invalid keyword "3".
>
> Cheers,
> Pieter
>
>
>> On 28 Jul 2017, at 20:23, Acee Lindem (acee) <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>
>>
>> From: "Joe Clarke (jclarke)" <[email-masked] <mailto:[email-masked]>>
>> Date: Friday, July 28, 2017 at 2:12 PM
>> To: Acee Lindem <[email-masked] <mailto:[email-masked]>>
>> Cc: "Pieter Lewyllie (pilewyll)" <[email-masked] <mailto:[email-masked]>>, "Benoit Claise (bclaise)" <[email-masked] <mailto:[email-masked]>>, "Mahesh Jethanandani (mahesh)" <[email-masked] <mailto:[email-masked]>>, RTG YANG Design Team <[email-masked] <mailto:[email-masked]>>, Ing-Wher Chen <[email-masked] <mailto:[email-masked]>>, "Carl Moberg (camoberg)" <[email-masked] <mailto:[email-masked]>>, YANG Doctors <[email-masked] <mailto:[email-masked]>>, Henrik Levkowetz <[email-masked] <mailto:[email-masked]>>, Radek Krejčí <[email-masked] <mailto:[email-masked]>>
>> Subject: Re: YANG Tool Requests => regex validator
>>
>>
>>         https://yangcatalog.org/yangre doesn’t work for me, no matter how I quote
>>         it.
>>
>>         I tried with the inet:ipv4-address pattern and I did get this to work with
>>         W3CGREP but not yangre no matter how I quoted it. For W3CGREP, I had to
>>         remove all the quoting and concatenation. This could be error prone for a
>>         very complex pattern.
>>
>>     Hmmm. This pattern from ietf-inet-types (note: I had the single quotes in the UI):
>>
>>     '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?'
>>
>>     Works for me with this string (only with YANGRE):
>>
>>     192.168.1.1
>>
>> Nope – exact same pattern and test string fails for me. If I unquote it, it works with W3CGREP. 
>>
>>
>> Thanks,
>> Acee 
>>
>>
>>
>>     Joe
>>
>

-- 
Radek Krejci
mobile  : +420 732 212 714
office  : +420 234 680 256
e-mail  : [email-masked]
LinkedIn: http://www.linkedin.com/in/radekkrejci

CESNET, Association of Legal Entities
Zikova 4
160 00 Praha 6
Czech Republic

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Tue, 1 Aug 2017 12:57:54 +0000

Hi Radek, 

So then if one is to cut and paste that quoted pattern from the IETF yang
model into https://yangcatalog.org/yangre, it should work? Most complex
patterns are multi-line quoted strings for readability in Internet drafts.
Will these multi-line patterns be handled correctly or should on do the
concatenation manually after cut and past?

Thanks,
Acee 

On 8/1/17, 5:30 AM, "Radek Krejčí" <[email-masked]> wrote:

>Hi,
>yangre requires patterns as you would write them in a YANG schema. Using
>{ in pattern without quoting is illegal - see RFC 7950, sec 6.1.3. It is
>the same issue as have semicolon there. So, yangre actually does not fail
>with quantifier, but since it follows {, a valid YANG statement is
>expected, but 3 is an invalid statement/keyword.
>
>yangre actually takes the pattern, creates very small YANG schema and
>then 1) validates the schema (which does not pass in this case) and then
>2) checks if the leaf of string type with the specified pattern(s) can
>have the provided value (if it matches the patterns). So it actually
>checks what yanglint does in case the string is specified as leaf's
>default value.
>
>Regarding the quotes in CLI - it is quite challenging to make quotation
>correct because of interpreting quotation by shell. See help, there are
>some examples but there can be some special cases or different behavior
>with different shell. The current yangre accepts alternative input via a
>file (patterns on separate lines, leading space for invert match, at the
>last line after a single blank line the string for matching) which would
>avoid issues with quoting. Anyway, as we were discussing with Pieter -
>Python seems to do the quotation correct (or probably just bypass the
>shell), but yangre expect the patterns _including_ the quotation as you
>would write it in YANG schema. Only this way all the YANG rules can be
>checked.
>
>Regards,
>Radek
>
>> (([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3} matches
>>192.1.1. on W3CGREP, but not on YANGRE ( yangre error: Invalid keyword
>>"3”. )
>>
>> '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}’ matches
>>192.1.1. on YANGRE, but not on W3CGREP (which is expecting the ‘ in the
>>test string)
>>
>> The same behaviour can be observed in the CLI, you just have to add
>>wrap the expressions with appropriate quotes.
>>
>> *Radek,* do you have an idea why the YANGRE fails with the quantifier?
>>Here are a couple of CLI examples that I observed:
>> yangre -p 
>>"(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9
>>]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?" "192.168.0.1"
>> /yangre -p 
>>"'(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-
>>9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?'" “192.168.0.1”
>> matches (no output)
>>
>> yangre -p 
>>'"(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-
>>9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?"' "192.168.0.1"
>> yangre error: Invalid keyword
>>"(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9
>>]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?”".
>>
>> yangre -p 
>>'(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9
>>]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?' "192.168.0.1"
>>> Cc: "Pieter Lewyllie (pilewyll)" <[email-masked]
>>><mailto:[email-masked]>>, "Benoit Claise (bclaise)"
>>><[email-masked] <mailto:[email-masked]>>, "Mahesh Jethanandani
>>>(mahesh)" <[email-masked] <mailto:[email-masked]>>, RTG YANG Design
>>>Team <[email-masked] <mailto:[email-masked]>>,
>>>Ing-Wher Chen <[email-masked]
>>><mailto:[email-masked]>>, "Carl Moberg (camoberg)"
>>><[email-masked] <mailto:[email-masked]>>, YANG Doctors
>>><[email-masked] <mailto:[email-masked]>>, Henrik
>>>Levkowetz <[email-masked] <mailto:[email-masked]>>, Radek
>>>Krejčí <[email-masked] <mailto:[email-masked]>>
>>>'(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-
>>>9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?'
>>>

---

From: Radek Krejčí <[email-masked]>
Date: Tue, 1 Aug 2017 15:20:58 +0200

Hi Acee,
if you just remove newlines (putting multiple lines on a single line) and preserve +, it works just fine even in GUI.Just try "[0-9]*" + "[a-z]*" with e.g. 01ac test string. I've tried to copy multi-line pattern into GUI's pattern field and it seems to be removing newline characters automatically, so you should be straightforward.

yangre on cli (with command line patterns, not when reading patterns from file) accepts even multi-line patterns (when correctly provided to shell):

$ yangre -V -p '"[0-9]*" +
"[a-z]*"' 1342dsga
pattern  1: "[0-9]*" +
"[a-z]*"
matching 1: regular
string    : 1342dsga
result    : matching

Regards,
Radek