ure: add groups, span, start, end and sub methods - #3815

Thanks for the feedback @dmazzella . I updated this PR with your suggested changes.

Except, I didn't remove the \g<number> pattern. Reading the Python docs, this "g" pattern allows two things that standard \number doesn't: it allows to reference the entire group with \g<0> (\0 is not allowed); and it allows to have integers after the replacement, like \g<1>2 works but you can't do that with \12.

So "g" is more general than just a plain integer, but for most cases a plain integer is enough and you don't want tho hassle of writing out the "g" and the angle brackets. Maybe there is a way to allow both with minimal code size...

Except, I didn't remove the \g pattern. Reading the Python docs, this "g" pattern allows two things that standard \number doesn't: it allows to reference the entire group with \g<0> (\0 is not allowed); and it allows to have integers after the replacement, like \g<1>2 works but you can't do that with \12.

If add this check at https://github.com/dpgeorge/micropython/blob/36f45ced8608822b8c22ea6907e52bb4658a6596/extmod/modure.c#L345:

if (match_no == 0 && is_group_number == 0) {
    memmove((char *)(start_group), (char *)(end_group), strlen(start_group));
    continue;
}

fix \0 replace and enable \g<0>

Hi Damian,
your last commit work in most of normal use case, but as python docs:

\number
Matches the contents of the group of the same number. Groups are numbered starting from 1. For example, (.+) \1 matches 'the the' or '55 55', but not 'thethe' (note the space after the group). This special sequence can only be used to match one of the first 99 groups. If the first digit of number is 0, or number is 3 octal digits long, it will not be interpreted as a group match, but as the character with octal value number. Inside the '[' and ']' of a character class, all numeric escapes are treated as characters.

the special \0 group match must be considered.

the special \0 group match must be considered.

Yes. But for now I just wanted to simplify the existing code to make it more compact, without changing the functionality (at least not changing it too much).

or number is 3 octal digits long, it will not be interpreted as a group match, but as the character with octal value number.

I'm not sure we can support octal number escapes here...

Hi Damien,

this patch octal_number_support.zip implements all needs by:

\number
Matches the contents of the group of the same number. Groups are numbered starting from 1. For example, (.+) \1 matches 'the the' or '55 55', but not 'thethe' (note the space after the group). This special sequence can only be used to match one of the first 99 groups. If the first digit of number is 0, or number is 3 octal digits long, it will not be interpreted as a group match, but as the character with octal value number

this is a little test for these features:

s = re.compile(
    '(calzino) (blu|bianco|verde) e (scarpa) (blu|bianco|verde)'
).sub(
    r'\g<1> colore \2 con \g<3> colore \4? ...\0 -\01 -\05 - \g<0> - \01323 - \0898877665544332211 - \26796576576576576576576576576576',
    'calzino blu e scarpa verde'
)
print(s.encode())
print(s)

the output for upy and cpy are identicals.

Hi @dpgeorge,
did you have time to check the last patch? Any suggestions?

did you have time to check the last patch? Any suggestions?

I had a look at it, the patch that adds support for octal numbers. There's quite a lot of code added for this feature which is probably rarely used.

And you can already achieve the same thing (inserting arbitrary bytes) by using the non-r form of a string. For example, if you want to do this:

re.sub(r'(123)', r'456\072\1', 'abc123def')

then you can instead write it like this, which doesn't require support for interpreting octals in the regex code:

re.sub(r'(123)', '456\072\\1', 'abc123def')

(note the subtle change: "r" is gone in the 2nd string and there's a double backslash before the "1")

So I'd say we shouldn't add octal support, to keep code size down.

@dpgeorge the only modification i can suggest is that the special \0 group match must be considered.

the special \0 group match must be considered.

In what way do you mean? At the moment \0 and \<0> are the same and will output the entire match.

In what way do you mean? At the moment \0 and <0> are the same and will output the entire match.

something like that:

// Group specified with syntax "\number"
if (!is_g_format && *repl == '0') {
    vstr_add_byte(vstr_return, 0);
    repl++;
    continue;
}

for test:
print(re.sub(r'(123)', '456\072\\1\\01\\g<0>', 'abc123def').encode())

the output in cPy:
b'abc456:123\x01123def'

the output in uPy:
b'abc456:123\x001123def'

or removing:
vstr_add_byte(vstr_return, 0);

the output in uPy:
b'abc456:1231123def'

@dpgeorge

with the current implementation there is a problem of "Segmentation fault" in the construction of the number from the string.

with a group like:
print(re.sub(r'(123)', '456\072\\199999999999999999999999999999999999999999999\\01\\g<0>', 'abc123def').encode())

to avoid this I suggest adding control over the maximum of acceptable groups (99) like this:
} while ('0' <= *repl && *repl <= '9' && match_no * 10 + (*repl - '0') <= 99);

cPy:
sre_constants.error: invalid group reference 19 at position 5

in uPy we can add:

if (match_no < match->num_matches) {
...
} else {
    mp_raise_ValueError("invalid group reference");
}

something like that: ...

I don't think there's a need to support \0 to output null (or nothing). None of these behaviours will match CPython properly without substantial additional code. I think \0 doing the same thing as \<0> is good enough behaviour. At the very least there is a subset of behaviour of re/ure that is common to CPython and uPy. That's the important thing (so scripts can be written that run the same under both).

with the current implementation there is a problem of "Segmentation fault" in the construction of the number from the string.

Thanks for pointing this out, I've now fixed it in the latest set of commits in this PR. Simplest solution was to make match_no an unsigned int.

I have tried this but the error differ from cPy and uPy.

cPy refer to group number 19
uPy to 4294967295

use of something like:
} while ('0' <= *repl && *repl <= '9'&& match_no < match->num_matches);
or
} while ('0' <= *repl && *repl <= '9' && match_no * 10 + (*repl - '0') <= 99);

produce the same group number.

4294967295 is 0xffffffff

I have tried this but the error differ from cPy and uPy.

cPy refer to group number 19
uPy to 4294967295

IMO that's not an issue. At least uPy raises an error in this case, and doesn't error out when there's not an error.

I further optimised this set of commits for space and heap usage, updated docs, and merged in 1f86460 through 4727bd1

Thanks @dmazzella for the original patch for these features!