Fatal process out of memory: DateTimePatternGeneratorCache::CreateGenerator

[Angelo Dell'Aera]

Hi all,

starting from a few weeks I started observing these crashes while using the Python

wrapper STPyV8 [1]. I am the lead developer of STPyV8 and no relevant changes were

introduced recently other than updating the V8 version to use in the building process. 

This trace was generated using V8 12.2.281.19

<--- Last few GCs --->

[380775:0x17d0000]   254778 ms: Scavenge 75.3 (80.3) -> 73.4 (80.3) MB, pooled: 0 MB, 0.31 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;
[380775:0x17d0000]   254783 ms: Scavenge 75.3 (80.3) -> 73.4 (80.3) MB, pooled: 0 MB, 0.31 / 0.00 ms  (average mu = 1.000, current mu = 1.000) allocation failure;
[380775:0x17d0000]   265747 ms: Scavenge 76.3 (81.5) -> 75.1 (79.0) MB, pooled: 2 MB, 0.88 / 0.05 ms  (average mu = 1.000, current mu = 1.000) allocation failure;


<--- JS stacktrace --->



#
# Fatal process out of memory: DateTimePatternGeneratorCache::CreateGenerator
#
==== C stack trace ===============================

    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(v8::base::debug::StackTrace::StackTrace()+0x13) [0x7f7d3ef16503]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(+0xcf419b) [0x7f7d3ef1819b]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(v8::base::FatalOOM(v8::base::OOMType, char const*)+0x38) [0x7f7d3f879f68]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&)+0x2ac) [0x7f7d3ef1b58c]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(+0xfe498a) [0x7f7d3f20898a]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(v8::internal::JSDateTimeFormat::CreateDateTimeFormat(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::JSDateTimeFormat::RequiredOption, v8::internal::JSDateTimeFormat::DefaultsOption, char const*)+0x8b6) [0x7f7d3f206e06]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(v8::internal::JSDateTimeFormat::New(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, char const*)+0x1a) [0x7f7d3f207efa]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(v8::internal::Builtin_DateTimeFormatConstructor(int, unsigned long*, v8::internal::Isolate*)+0xfe) [0x7f7d3f8bed4e]
    /opt/env/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so(+0x151e5b6) [0x7f7d3f7425b6]

Regards,

Angelo

[1] https://github.com/cloudflare/stpyv8

[Ben Noordhuis]

Did you also update ICU? That error message indicates V8 couldn't
create a icu::DateTimePatternGenerator instance.

Try building V8 with debug checks enabled, that should give you a
better idea of the root cause.

[Angelo Dell'Aera]

Did you also update ICU? That error message indicates V8 couldn't
create a icu::DateTimePatternGenerator instance.

The Github Actions workflow I use to build STPyV8 automatically fetches the V8 repository, 

perfoms a checkout of the V8 version I specify in my settings, syncs the code with gclient 

sync and builds it using the following gn args

gn_args = {
    "dcheck_always_on": "false",
    "is_component_build": "false",
    "is_debug": "true" if os.environ.get("STPYV8_DEBUG") else "false",
    "treat_warnings_as_errors": "false",
    "use_custom_libcxx": "false",
    "v8_deprecation_warnings": "true",
    "v8_enable_disassembler": "false",
    "v8_enable_i18n_support": "true",
    "v8_enable_pointer_compression": "false",
    "v8_enable_31bit_smis_on_64bit_arch": "false",
    "v8_imminent_deprecation_warnings": "true",
    "v8_monolithic": "true",
    "v8_use_external_startup_data": "false",
}

No manual ICU update then.

 

Try building V8 with debug checks enabled, that should give you a
better idea of the root cause.

I tried but all I got is this error

*** stack smashing detected ***: terminated 

I perfomed an additional test rebuilding setting v8_enable_i18n_support to false and the

issue disappeared. 

Please be aware that I am able to reproduce the reported issue very reliably on Debian 11 

but everything works fine if the same code runs on Gentoo Linux, macOS and Windows.

Regards,

Angelo

[Ben Noordhuis]

I'd try to narrow down why that debug build crashes the way it does.
What does the backtrace look like in gdb? What happens in a debug +
ASAN or MSAN build?

Another thing you could try for debugging is to set
use_custom_libcxx=true and see if that makes things worse or better.

[Angelo Dell'Aera]

I collected a gdb backtrace and it shows something which is potentially interesting

(gdb) info stack
#0  0x00007ffff4d72ae6 in v8::base::OS::Abort() () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#1  0x00007ffff4d6af7c in v8::base::FatalOOM(v8::base::OOMType, char const*) () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#2  0x00007ffff440c58c in v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#3  0x00007ffff46f998a in v8::internal::(anonymous namespace)::DateTimePatternGeneratorCache::CreateGenerator(v8::internal::Isolate*, icu_73::Locale const&) () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#4  0x00007ffff46f7e06 in v8::internal::JSDateTimeFormat::CreateDateTimeFormat(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::JSDateTimeFormat::RequiredOption, v8::internal::JSDateTimeFormat::DefaultsOption, char const*) () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#5  0x00007ffff46f8efa in v8::internal::JSDateTimeFormat::New(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, char const*) ()
   from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#6  0x00007ffff4dafd4e in v8::internal::Builtin_DateTimeFormatConstructor(int, unsigned long*, v8::internal::Isolate*) () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#7  0x00007ffff4c335b6 in Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#8  0x00007ffff4b9f3fc in Builtins_InterpreterPushArgsThenFastConstructFunction () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#9  0x00002136d010c8f9 in ?? ()
#10 0x00002136d010c8f9 in ?? ()
#11 0x0000000500000000 in ?? ()
#12 0x000027e541540c19 in ?? ()
#13 0x000027e541540c19 in ?? ()
#14 0x000027e541540c19 in ?? ()
#15 0x00001bbc90f9d349 in ?? ()
#16 0x0000000000000030 in ?? ()
#17 0x00007fffffff7940 in ?? ()
#18 0x00007ffff4d2736c in Builtins_ConstructHandler () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
#19 0x0000207ba8729fe1 in ?? ()
#20 0x00001bbc90fb50b9 in ?? ()
#21 0x000027e541540061 in ?? ()
#22 0xfffffffffffffffe in ?? ()
#23 0x00007ffff4d1b0a0 in Builtins_SetNamedPropertyHandler () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

As we can see from #3

#3  0x00007ffff46f998a in v8::internal::(anonymous namespace)::DateTimePatternGeneratorCache::CreateGenerator(v8::internal::Isolate*, icu_73::Locale const&) () from /home/angelo/.local/lib/python3.9/site-packages/_STPyV8.cpython-39-x86_64-linux-gnu.so

the CreatePattern second parameter type is supposed to be icu_73::Local.

As I mentioned in a previous email, Debian 11 is the only environment that allows me to reproduce the issue reliably while the

issue is not present in all the other environments. So I checked the system libicu version and realized that libicu 67 is installed

on Debian 11 and libicu 74 is installed on all the other systems.

During the build workflow, I statically link libv8_monolith to STPyV8 so I don't really expect but if V8 ends up using the system

libicu (because I do not link statically the third party library too? is it really required?) and that libicu 73 API is not compatible

with the libicu 67 one that could explain what I am observing.

Angelo

[Ben Noordhuis]

I think you've found the cause. (I had a hunch it was conflicting
symbols/libraries, hence my suggestion of turning on
use_custom_libcxx.)

It's possible something else also loads (or exports a copy of) ICU and
makes you end up with the wrong version when the dynamic linker does
runtime linking.

[Angelo Dell'Aera]

Yet another test confirms my assumption. I performed the same tests on a Debian 12 host

(libicu 72) and no issues at all. I am going to try to figure out if the issue can be fixed some

way with some linking hack.

Thank you very much for the support. Really appreciated it.

Angelo

[Angelo Dell'Aera]

It seems like I figured out the issue behind that and it is related to ICU.

From the ICU 74.2 release note [1]

Important: DateFormat.getInstanceForSkeleton() and the DateTimePatternGenerator sometimes used the wrong patterns because they failed to use/inherit certain data (ICU-22575 — CLDR 44 had removed some redundant data that ICU relied on)

This looks exactly what I am observing so maybe fixing the issue is just a matter

of updating the third party library ICU to version 74.2.

[1] https://github.com/unicode-org/icu/releases/tag/release-74-2