Multi-line ^$ anchors in Perl mode

[Vladimir Gladkov]

In the default Perl mode, it's impossible to configure RE2::Options for the multi-line ^$ anchors. The only way to enable multi-line anchors in Perl mode is to prepend all relevant patterns with (?m), which is (1) rather inconvenient (2) less efficient than configuring that via RE2::Options, and (3) inconsistent with other configuration options (longest-match, case-insensitivity, etc.)

Is there any strong reasoning behind this design?

If not, then I would like to propose trivial modifications to RE2::Options to fix this limitation. Of course, we can't change the semantics of RE2::Options::set_one_line() to maintain full backward compatibility, but we can introduce a new 3-state (undefined, true, false) setting RE2::Options::set_multi_line() which would supersede the old one-line setting and, if true, clear the Regexp::ParseFlags::OneLine before returning flags from RE2::Options::ParseFlags().

Thoughts?

[Paul Wankadia]

On Wed, Nov 1, 2023 at 9:44 PM Vladimir Gladkov <vov...@gmail.com> wrote:

In the default Perl mode, it's impossible to configure RE2::Options for the multi-line ^$ anchors. The only way to enable multi-line anchors in Perl mode is to prepend all relevant patterns with (?m), which is (1) rather inconvenient (2) less efficient than configuring that via RE2::Options, and (3) inconsistent with other configuration options (longest-match, case-insensitivity, etc.)

Is there any strong reasoning behind this design?

This came up in https://groups.google.com/g/re2-dev/c/_1vmeAgH3hM/ a few years ago, but my position hasn't changed since then because, well, despite my continued desire for a consistent interface, trying to improve this particular aspect of the interface is too much risk for too little reward. :(

[Russ Cox]

On Wed, Nov 1, 2023 at 6:44 AM Vladimir Gladkov <vov...@gmail.com> wrote:

In the default Perl mode, it's impossible to configure RE2::Options for the multi-line ^$ anchors. The only way to enable multi-line anchors in Perl mode is to prepend all relevant patterns with (?m), which is (1) rather inconvenient (2) less efficient than configuring that via RE2::Options,

Neither of these is really true. It's trivial to write "(?m)"+s, and RE2 will parse those extra 4 bytes quite efficiently.

Best,

Russ

[Vladimir Gladkov]

(1) rather inconvenient (2) less efficient than configuring that via RE2::Options,

Neither of these is really true. It's trivial to write "(?m)"+s, and RE2 will parse those extra 4 bytes quite efficiently.

(1) Convenience is subjective, of course, but as the discussion mentioned by Paul shows, other ppl besides myself also had stumbled over this particular RE2 idiosyncrasy. Yes, now it's described in the comment, and the situation has improved. However, still -- on the one hand, we have a way to configure the multi-line mode intuitively and consistently with all the other options (you don't even have to remember anything -- you can see all available options in the member drop-down list in your IDE). On the other hand, you need to remember an obscure prefix (or look it up) -- most ppl know basic regex syntax, but how many know (?m) by heart? Also, you must remember to prepend this prefix to all patterns in your program. I bet most ppl would think the first approach is more convenient.

(2) OK, I should have used the "slightly" qualifier ;) But again -- we either specify parse flags directly and once (then share the options). Or strcat the (?m) prefix to each user pattern and then parse this prefix to extract the same parse flags -- again, once per pattern. Of course, this is not performance-critical code, but I'm sure you would prefer the first approach over the second one efficiency-wise.

This came up in https://groups.google.com/g/re2-dev/c/_1vmeAgH3hM/ a few years ago, but my position hasn't changed since then because, well, despite my continued desire for a consistent interface, trying to improve this particular aspect of the interface is too much risk for too little reward. :(

I fully agree that the reward is not that big -- this is not a critical issue to any extent. But if the only reason holding us back from fixing it is the risk of breaking things, then there's no risk and guaranteed 100% backward compatibility. As I said in the original message, we don't touch the semantics of one_line. Instead, we introduce an additional setting (naming is discussible -- multi_line/one_line_new/one_line_ex/one_line_override/etc). When undefined, it does nothing; when defined, it overrides one_line. Done!

Once again, this is not a critical issue, so I don't insist. Just a small, simple, and safe improvement of RE2 API usability.

[Vladimir Gladkov]

Now, where all these patches & propositions are coming from? I'm currently working on a modification of RE2 to enable (1) matching over rewindable streams (e.g., running a regexp over huge files that can't be mapped all at once, live TCP streams, etc.) and (2) regexp switches / lexers. Somewhat similar to Hyperscan -- but in a compact cross-platform library that is RE2. These modifications are not going to break any existing APIs or RE2-dependent code -- they'll introduce a new matcher interface just like RE2::Set does. During work, a few things caught my eye, so I'm proposing simple and safe ones to fix. Hope this makes sense.

[Paul Wankadia]

On Thu, Nov 2, 2023 at 2:53 PM Vladimir Gladkov <vov...@gmail.com> wrote:

I fully agree that the reward is not that big -- this is not a critical issue to any extent. But if the only reason holding us back from fixing it is the risk of breaking things, then there's no risk and guaranteed 100% backward compatibility. As I said in the original message, we don't touch the semantics of one_line. Instead, we introduce an additional setting (naming is discussible -- multi_line/one_line_new/one_line_ex/one_line_override/etc). When undefined, it does nothing; when defined, it overrides one_line. Done!

The desired end state is to have one knob that works, not to have one knob that doesn't work and one trinary knob that does work iff the user happens to know about it. The problem is having to make a breaking change in order to get there from here (otherwise, the API surface overall becomes more complex instead of less complex) and lacking a strong reason to make it.

On Thu, Nov 2, 2023 at 2:59 PM Vladimir Gladkov <vov...@gmail.com> wrote:

Now, where all these patches & propositions are coming from? I'm currently working on a modification of RE2 to enable (1) matching over rewindable streams (e.g., running a regexp over huge files that can't be mapped all at once, live TCP streams, etc.) and (2) regexp switches / lexers. Somewhat similar to Hyperscan -- but in a compact cross-platform library that is RE2. These modifications are not going to break any existing APIs or RE2-dependent code -- they'll introduce a new matcher interface just like RE2::Set does. During work, a few things caught my eye, so I'm proposing simple and safe ones to fix. Hope this makes sense.

Cool, that sounds fun. Ever since RE2 took a dependency on Abseil, supporting absl::Cord has shifted from "basically impossible" to "probably never going to happen", but I like to think about it every now and then. It would be more akin to Hyperscan's "vectored mode" rather than "streaming mode", so presumably not quite the same as your use case anyway.