OOM and AddNearHeapLimitCallback

[Prakash Bailkeri]

Hello V8-Experts,

I am working on a project where I need to run untrusted user provided javascript. I am worried about faulty user script bringing down the entire application by hogging memory. So, idea is to implement budgeting of heap memory for each javascript.

I looked at different projects that tried to solve the process exit due to OOM error. Few ideas that collected are

       1. Running separate thread and RequestInterrupt. In interrupt routine, using GetHeapStatistics to check heap usage. 

           https://github.com/phpv8/v8js/blob/php7/v8js_timer.cc#L36:13

       2. Using AddGCEpilogueCallback and GetHeapStatistics

  https://github.com/discourse/mini_racer/blob/master/ext/mini_racer_extension/mini_racer_extension.cc#L194:13

I am worried about the overhead and performance impact of these approaches.

Recently, following commit introduced "AddNearHeapLimitCallback".

Bug: https://bugs.chromium.org/p/chromium/issues/detail?id=824214 

Commit: https://chromium.googlesource.com/v8/v8/+/84a80e103b090f35334fe2d168ece17489395417

I tried using this API in my sample program to detect OOM and dispose the isolate executing faulty script. [Please see below for the sample program]

My experiment detected the OOM error and terminated the faulty script. 

I have few questions regarding this API:

1. Are there any caveats in using this API? How reliably it detects the OOM condition? Are there scenarios in which it will fail to detect the OOM?

2. In the sample code below, in function, "MyNearHeapLimitCallback": I am terminating the execution and returning the new heap limit by adding 5MB. If I return the initial_heap_limit as is (i.e. without adding 5 MB), the faulty script will make my application crash with OOM. Why?

3. When the AddNearHeapLimitCallback is being called, I assume that the javascript execution is paused. Is this a correct understanding?

Thanks, Prakash

==== Sample program ==

size_t MyNearHeapLimitCallback(void* data, size_t current_heap_limit,

                             size_t initial_heap_limit) {

  v8::Isolate *isolate = (v8::Isolate *)data;

  isolate->TerminateExecution();

  return initial_heap_limit + 5 * 1024 * 1024;

}

int main(int argc, char* argv[]) {

  v8::V8::InitializeICUDefaultLocation(argv[0]);

  v8::V8::InitializeExternalStartupData(argv[0]);

  std::unique_ptr<v8::Platform> platform = v8::platform::NewDefaultPlatform();

  v8::V8::InitializePlatform(platform.get());

  v8::V8::Initialize();

  while(true) {

      v8::Isolate::CreateParams create_params;

      create_params.array_buffer_allocator = v8::ArrayBuffer::Allocator::NewDefaultAllocator();

      v8::ResourceConstraints constraints;

      constraints.set_max_old_space_size(10);

      create_params.constraints = constraints;

      v8::Isolate* isolate = v8::Isolate::New(create_params);

      isolate->AddNearHeapLimitCallback(MyNearHeapLimitCallback, isolate);

      {

          v8::Isolate::Scope isolate_scope(isolate);

          v8::HandleScope handle_scope(isolate);

          v8::Local<v8::Context> context = v8::Context::New(isolate);

          v8::Context::Scope context_scope(context);

          v8::Local<v8::String> source =

              v8::String::NewFromUtf8(isolate, "a = []; for (;;) { a.push('hello'); }", 

                                      v8::NewStringType::kNormal)

              .ToLocalChecked();

          v8::Local<v8::Script> script = v8::Script::Compile(context, source).ToLocalChecked();

          v8::TryCatch try_catch(isolate);

          v8::MaybeLocal<v8::Value> result;

          result = script->Run(context);

          if (try_catch.HasCaught() && try_catch.HasTerminated()) {

              isolate->CancelTerminateExecution();

          }

      }

      delete create_params.array_buffer_allocator;

      isolate->Dispose();

  }

  v8::V8::Dispose();

  v8::V8::ShutdownPlatform();

  return 0;

}

[Zac Hansen]

Every time this question is asked, the answer always seems to be "run it in a separate process space."   There is no way to reliably protect / detect and recover from memory issues in V8.  There's usually a "patches welcome as long as it has no affect on performance" at the end.

[mog...@syntheticsemantics.com]

Hi Prakash,

I recommend taking a look at Laverdet's approach used in Node.js:  https://github.com/laverdet/isolated-vm

                       -J

On Monday, June 4, 2018 at 3:35:58 AM UTC-7, Prakash Bailkeri wrote:

[Ben Noordhuis]

On Tue, Jun 5, 2018 at 6:12 AM, <mog...@syntheticsemantics.com> wrote:
> I recommend taking a look at Laverdet's approach used in Node.js:
> https://github.com/laverdet/isolated-vm

What isolated-vm does isn't 100% bulletproof either. I think the
advice of "run it in a separate process" still stands.

For the curious, isolated-vm uses a custom arraybuffer allocator that
tracks how much memory has been allocated and throws a termination
exception when a threshold is exceeded. There's a similar check in a
GC epilogue callback. That only covers some out-of-memory conditions
though, other allocation failures still kill the process.

[Prakash Bailkeri]

Thanks for the response.

Any advice on usage of "AddNearHeapLimitCallback"? [Please see my sample program]

Thanks, Prakash

--
--
v8-users mailing list
v8-u...@googlegroups.com
http://groups.google.com/group/v8-users
---
You received this message because you are subscribed to a topic in the Google Groups "v8-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/v8-users/qLs7-XT2Zvg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to v8-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ulan Degenbaev]

Hi Prakash,

I added the AddNearHeapLimitCallback API. The provided callback is invoked when V8's heap is close to the heap limit specified by the ResourceConstraints::max_old_space_size during V8 setup.

If the callback extends the heap limit by returning a number greater then the provided current_heap_limit, then the application will continue running until the new heap limit is reached. At that point the callback is invoked again. 

If the callback returns the current_heap_limit (or smaller number) then the application will crash with OOM if it continues allocating.

The initial_heap_limit is the original limit that V8 was set up with. It is provided for convenience, so that the callback can figure out if it has already extended the limit without maintaining state.

The callback is called on the same thread as the running JavaScript. So during callback invocation JavaScript is paused. 

Overall, I think the API should work for your case.

Caveats:

- the callback may be invoked multiple times.

- if the callback does not extend the heap limit, then the application is not guaranteed to OOM. For example, if the application stops allocating and never reaches the heap limit.

- if the heap limit is extended by small number, then the application may still crash with OOM. For example, if the application wants to allocate 10MB, but the limit is extended by 5MB.

- if V8 runs out of system memory without reaching the heap limit, then it will crash with OOM without invoking the callback. For example, if malloc/mmap fails there is no way to safely continue execution.

I hope that answers your questions. Please let me know if I missed anything.

Cheers,

Ulan.

On Tue, Jun 5, 2018 at 11:28 AM, Prakash Bailkeri <prakash....@gmail.com> wrote:

Thanks for the response.

Any advice on usage of "AddNearHeapLimitCallback"? [Please see my sample program]

Thanks, Prakash

--
--
v8-users mailing list
v8-u...@googlegroups.com
http://groups.google.com/group/v8-users
---

You received this message because you are subscribed to the Google Groups "v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+unsubscribe@googlegroups.com.

[Prakash Bailkeri]

Ulan, 

Thanks for the detailed answer. 

I have a question on the caveats you listed:

>> "if the heap limit is extended by small number, then the application may still crash with OOM. For example, if the application wants to allocate 10MB, but the limit is extended by 5MB." << 

Question 1: In the "AddNearHeapLimitCallback", I don't know what is the memory size requested. So, how can I safely increase the heap limit to avoid OOM?

Question 2: Even though, I plan to terminate the isolate execution(isolate->TerminateExecution()), I need to increase the heap limit to avoid crash.  In such a case(TerminateExecution), is there a way I can avoid crash without increasing the limit(say, by returning same or lesser memory limit as return code from MyNearHeapLimitCallback)?

Thanks, Prakash

To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+u...@googlegroups.com.

[ma...@jsonar.com]

Running in a separate process space doesn't work very well if you want any custom function callbacks you might have hooked into v8 to be able to access or manipulate data that might be used by the rest of your application, since each process has no access to your main application's data space.