Avoid signed overflow when deriving the forward Prog memory budget

[Suwit k]

Hi RE2 maintainers,

I found a small undefined-behavior edge in the public RE2::Options::set_max_mem() path and wanted to check the intended fix before proposing code.

With max_mem = INT64_MAX, RE2::Init() currently derives the forward-program budget using options_.max_mem() * 2 / 3. The multiplication overflows signed int64_t before the division. I reproduced this through the public API with Clang UBSan on current upstream (972a15cedd008d846f1a39b2e88ce48d7f166cbd).

A minimal arithmetic-only way to preserve floor(2*m/3) without the overflowing intermediate is to compute it from the quotient and remainder, for example (m / 3) * 2 + ((m % 3) * 2) / 3. I also have a regression case that constructs a trivial RE2 with max_mem = INT64_MAX.

I verified the local change with the regression test, an UBSan before/after reproducer, and the whole-tree GNU Make test suite; the latter completed with ALL TESTS PASSED.

Would that direction fit the intended semantics of max_mem, or would you prefer the option to be clamped or rejected at a different boundary?

I have kept the patch private while checking the contribution process and duplicate history. I can send the tested change for review after design confirmation.

For transparency, AI-assisted tooling was used during analysis; the reproducer, source trace, fix behavior, and test results above were independently verified on the referenced revision.