Debugging embedded V8 with ChromeDevTools

[Immanuel Haffner]

Hi all,

I followed this V8 documentation [1] to embed V8 into my application. The application compiles inputs to WebAssembly modules and executes them in embedded V8.  Now I would like to debug these WebAssembly modules.  Searching online brought me to ChromeDevTools and the v8-inspector API.  I mostly followed this guide [2] to set up a `V8InspectorSession` and implemented the `V8Inspector::Channel` interface by creating a TCP server, waiting for a connection, and then in `sendResponse()` and `sendNotification()` I simply send the message over the TCP socket.

To debug, I launch chromium and connect to localhost on the port where the TCP server was created. Sadly, this doesn't work. The `sendResponse()` and `sendNotification()` methods are never invoked.

Can someone please point me to a tutorial or give me some guidance?

Thanks,

Immanuel

[1] https://v8.dev/docs/embed

[2] https://hyperandroid.com/2020/02/12/v8-inspector-from-an-embedder-standpoint/

[Elmi Ahmadov]

Hi.

I have an example project how to use V8 inspector API: https://github.com/ahmadov/v8_inspector_example. Hope this would help.

Best regards

[Immanuel Haffner]

Dear Elmi,

thank you for your reply. I had a long look at your example. To be honest, I find it a bit bloatet.

Let me phrase in my own words what I have learned from your code:

1) I need a socket to receive messages from CDT and send responses back to CDT.

2) I must implement `v8_inspector::V8Inspector::Channel` to have `sendResponse()` and `sendNotification()` send the respective message over the socket to CDT.

3) I need an instance of `v8_inspector::V8InspectorClient`. I am not sure whether I must overwrite anything from this class in a subclass. I think the base class is actually fine...

4) I must use `v8_inspector::V8Inspector::create()` to create an instance of `v8_inspector::V8Inspector`. I believe this is where all the inspector logic is implemented...

5) Using the `V8Inspector` instance I create a `v8_inspector::V8InspectorSession` by calling `connect()`. Here I must pass a reference to an instance of my `v8_inspector::V8Inspector::Channel` subclass.

This was the basic setup. Now it's time to run the server loop.

6) In a detached thread, read from the socket and pass the message on by invoking `dispatchProtocolMessage()` on my instance of `v8_inspector::V8InspectorSession`.

When running my code, the TCP acceptor is set up and waiting for a client. I visit `localhost:<PORT>` with Chromium (the chrome-devtool:// links just won't work). Then my application executes the WebAssembly and the browser says the page isn't working (ERR_EMPTY_RESPONSE).

What am I missing?

[Elmi Ahmadov]

Dear Immanuel,

You've got the right idea from my example code. You need a websocket to create a communication channel between V8 Inspector and CDT.

The URL to debug the code is usually as follows and it works for me.

devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=IP:WebSocketPort

Without a minimal sample code it's difficult to say what you're missing.
You can also use Chrome DevTools Network Activity tab (you need to open another Inspector console, see the screenshot), just filter WS to see what happens between your server and CDT, which helped me a lot.

 

There is another Chrome built-in feature that helped me is Protocol Monitor, you can here how to enable it: https://chromedevtools.github.io/devtools-protocol/

[Immanuel Haffner]

Hi again,

I figured that the `chome-devtools://` part doesn't work in Chromium.  Using `devtools://` instead works. However, I get a segmentation violation. 

Here is the code I wrote to connect the V8 inspector:

    struct ServerChannel : v8_inspector::V8Inspector::Channel

    {

        private:

        sockpp::tcp_socket *socket_;

        public:

        ServerChannel() { }

        void set_socket(sockpp::tcp_socket *socket) { socket_ = socket; }

        private:

        void sendResponse(int, std::unique_ptr<v8_inspector::StringBuffer> message) override {

            auto v8_sv = message->string();

            socket_->write_n(v8_sv.characters8(), v8_sv.length());

        }

        void sendNotification(std::unique_ptr<v8_inspector::StringBuffer> message) override {

            auto v8_sv = message->string();

            socket_->write_n(v8_sv.characters8(), v8_sv.length());

        }

        void flushProtocolNotifications() override { }

    };

    struct MyV8InspectorClient : v8_inspector::V8InspectorClient

    {

        private:

        sockpp::tcp_acceptor acceptor_;

        std::unique_ptr<ServerChannel> channel_;

        std::unique_ptr<v8_inspector::V8Inspector> inspector_;

        std::unique_ptr<v8_inspector::V8InspectorSession> session_;

        int16_t port_;

        public:

        MyV8InspectorClient(int16_t port, v8::Isolate *isolate)

            : acceptor_(port)

            , port_(port)

        {

            if (not acceptor_)

                throw std::runtime_error("failed to initalize TCP server");

            channel_ = std::make_unique<ServerChannel>();

            inspector_ = v8_inspector::V8Inspector::create(isolate, this);

            std::string state("mutable");

            v8_inspector::StringView sv(reinterpret_cast<const uint8_t*>(state.c_str()), state.length());

            session_ = inspector_->connect(

                /* contextGroupId= */ 1,

                /* channel=        */ channel_.get(),

                /* state=          */ sv

            );

        }

        MyV8InspectorClient(const MyV8InspectorClient&) = delete;

        MyV8InspectorClient(MyV8InspectorClient&&) = default;

        ~MyV8InspectorClient() {

            session_.reset();

            inspector_.reset();

            channel_.reset();

        }

        void onMessage(std::string_view sv) {

            v8_inspector::StringView msg(reinterpret_cast<const uint8_t*>(sv.data()), sv.length());

            session_->dispatchProtocolMessage(msg);

        }

        void launch() {

            std::cout << "WebSocket based Inspector Agent started.\n"

                      << "Open the following link in your Chrome/Chromium browser:\n\n\t"

                      << "devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:" <<  port_

                      << '\n' << std::endl;

            auto socket = std::make_unique<sockpp::tcp_socket>(acceptor_.accept());

            if (not *socket)

                throw std::runtime_error("failed to establish connection to TCP client");

            channel_->set_socket(socket.get());

            auto thread = std::thread([this](std::unique_ptr<sockpp::tcp_socket> socket) {

                uint8_t buf[512];

                std::vector<uint8_t> msg;

                msg.reserve(512);

                std::size_t n;

                for (;;) {

                    std::cerr << "receiving bytes..." << std::endl;

                    while ((n = socket->read_n(buf, ARR_SIZE(buf))) > 0)

                        msg.insert(msg.end(), buf, buf + n); // append recently read bytes to msg

                    if (not msg.empty()) {

                        std::cerr << "the received message is " << msg.size() << " bytes" << std::endl;

                        std::string_view sv(reinterpret_cast<char*>(&msg[0]), msg.size());

                        this->onMessage(sv);

                        msg.clear();

                    }

                    std::this_thread::yield();

                }

            },

            std::move(socket));

            thread.detach();

        }

    };

It's mostly taken from your example and condensed into a single class.  Instead of boost, I use sockpp for socket programming.i

After creating a `v8::Isolate`, I do the following:

        inspector_ = std::make_unique<MyV8InspectorClient>(cdt_port, isolate_);

        inspector_->launch();

However, when connecting to the inspector it says "WebSocket disconnected" and the application crashes with a segmentation violation:

    AddressSanitizer:DEADLYSIGNAL

    =================================================================

    ==13072==ERROR: AddressSanitizer: SEGV on unknown address 0x1d580000a838 (pc 0x561c8173e6b4 bp 0x7fff75fa6430 sp 0x7fff75fa6430 T0)

    ==13072==The signal is caused by a READ memory access.

        #0 0x561c8173e6b4 in v8::Isolate::GetHeapProfiler() ../../../../../third-party/v8/v8/src/execution/isolate.h:1059:48

        #1 0x561c82d3fd77 in v8_inspector::V8HeapProfilerAgentImpl::stopTrackingHeapObjectsInternal() ../../../../../third-party/v8/v8/src/inspector/v8-heap-profiler-agent-impl.cc:316:14

        #2 0x561c82d3feb9 in v8_inspector::V8HeapProfilerAgentImpl::disable() ../../../../../third-party/v8/v8/src/inspector/v8-heap-profiler-agent-impl.cc:204:3

        #3 0x561c8208f095 in v8_inspector::V8InspectorSessionImpl::~V8InspectorSessionImpl() ../../../../../third-party/v8/v8/src/inspector/v8-inspector-session-impl.cc:151:24

        #4 0x561c8208f33d in v8_inspector::V8InspectorSessionImpl::~V8InspectorSessionImpl() ../../../../../third-party/v8/v8/src/inspector/v8-inspector-session-impl.cc:147:51

        #5 0x561c80ec9eff in std::default_delete<v8_inspector::V8InspectorSession>::operator()(v8_inspector::V8InspectorSession*) const /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:84:2

        #6 0x561c80ec9d3e in std::__uniq_ptr_impl<v8_inspector::V8InspectorSession, std::default_delete<v8_inspector::V8InspectorSession> >::reset(v8_inspector::V8InspectorSession*) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:181:4

        #7 0x561c80eca183 in std::unique_ptr<v8_inspector::V8InspectorSession, std::default_delete<v8_inspector::V8InspectorSession> >::reset(v8_inspector::V8InspectorSession*) /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:455:7

        #8 0x561c80ec7193 in db::v8_helper::MyV8InspectorClient::~MyV8InspectorClient() /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/backend/V8Platform.hpp:74:18

        #9 0x561c80ec723b in db::v8_helper::MyV8InspectorClient::~MyV8InspectorClient() /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/backend/V8Platform.hpp:73:28

        #10 0x561c80eb361f in std::default_delete<db::v8_helper::MyV8InspectorClient>::operator()(db::v8_helper::MyV8InspectorClient*) const /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:84:2

        #11 0x561c80eaede6 in std::unique_ptr<db::v8_helper::MyV8InspectorClient, std::default_delete<db::v8_helper::MyV8InspectorClient> >::~unique_ptr() /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:360:4

        #12 0x561c80ea6e97 in db::V8Platform::~V8Platform() /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/backend/V8Platform.cpp:221:1

        #13 0x561c80ea6f2b in db::V8Platform::~V8Platform() /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/backend/V8Platform.cpp:218:1

        #14 0x561c80e9ea3f in std::default_delete<db::WasmPlatform>::operator()(db::WasmPlatform*) const /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:84:2

        #15 0x561c80e9e8f6 in std::unique_ptr<db::WasmPlatform, std::default_delete<db::WasmPlatform> >::~unique_ptr() /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:360:4

        #16 0x561c80e8efbd in db::WasmBackend::~WasmBackend() /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/backend/WebAssembly.hpp:110:8

        #17 0x561c80e8efeb in db::WasmBackend::~WasmBackend() /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/backend/WebAssembly.hpp:110:8

        #18 0x561c80ccf1ff in std::default_delete<db::Backend>::operator()(db::Backend*) const /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:84:2

        #19 0x561c80ca8b16 in std::unique_ptr<db::Backend, std::default_delete<db::Backend> >::~unique_ptr() /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unique_ptr.h:360:4

        #20 0x561c80c86dd9 in process_stream(std::istream&, char const*, db::Diagnostic) /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/shell.cpp:151:13

        #21 0x561c80c9a265 in main /home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/../../src/shell.cpp:675:13

        #22 0x7f6259a37001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)

        #23 0x561c80ba722d in _start (/home/immanuel/Documents/Work/PhD/mutable/mutable/build/debug/bin/shell+0x117222d)

    AddressSanitizer can not provide additional info.

    SUMMARY: AddressSanitizer: SEGV ../../../../../third-party/v8/v8/src/execution/isolate.h:1059:48 in v8::Isolate::GetHeapProfiler()

Can you spot what's going wrong?  I did not override `V8InspectorClientImpl::runMessageLoopOnPause` like you did. Is this necessary?

Kind regards,

Immanuel

[Immanuel Haffner]

Ok, I figured the cause for the segmentation violation. I must dispose of the inspector and the session and channel before disposing of the isolate.  I am still left with "WebSocket disconnected". The CDT Network monitor says "Connection closed before receiving a handshake response".  Maybe I did something wrong establishing a TCP connection?

[Elmi Ahmadov]

I checked your code and you're missing `runMessageLoopOnMessage`/`quitMessageLoopOnPause` and `schedulePauseOnNextStatement` (this could be redundant), but in general, these methods are necessary. in `runMessageLoopOnPause` method, you should block the thread until there is a new message from CDT, in case there is a breakpoint and CDT requires user action.
Maybe your code does not wait for a user action and execution is done very quickly.

Could you please post WS messages from CDT Network monitor tab? There should be such response messages: `Debugger.scriptExecuted` and `Debugger.paused`.
And one important thing is you have to execute your code after you got a `Runtime.runIfWaitingForDebugger` message from CDT.

[Immanuel Haffner]

Hi Elmi,

thank you for checking. My biggest mistage was to write directly to a socket rather than use the WebSocket protocol. After doing so, i now see some messages in CDT. However, there are only 8 green and one last red message. I suppose the last message wasn't sent in full. The second last message was "Runtime.runIfWaitingForDebugger". There is no `Debugger.scriptExecuted` message, though.

Checking V8 code I couldn't find `schedulePauseOnNextStatement`, who is invoking that?

How do I attach an image to a Google Groups post?

[Elmi Ahmadov]

Seems you've managed to make a communication between CDT and your server. I also checked `sockpp` which does not support websocket protocol out of box. Probably, you should implement all required parts yourself.

I'm using Google Groups old UI which allows me to add image to the post, it's next to the `Link`:

But I also couldn't find it in the new UI, if you're using the new UI you can switch back to the old UI, click settings (gear icon) at the top of the page and click "Return to classic Google Groups", then you can attach an image.

[Immanuel Haffner]

Here is what the network messages in CDT look like:

I guess the execution doesn't wait for CDT. My WebAssembly function is executed right away and the connection is closed.

How do I make my `v8::Isolate` instance wait for commands from CDT?

[Elmi Ahmadov]

Hi Immanuel,

Seems the messages that come from V8 instance are cut off, maybe because of sockpp. They are also not complete, there should be a response for each request but in your case there are only 3 responses.

The logic to wait the V8 instance should be done in `runMessageLoopOnPause` method.

In my example, I'm calling a Boost method to get a new message from CDT. So, until there is a new message the V8 instance would wait.

[Immanuel Haffner]

Hi Elmi,

I am using boost web sockets now, just like you do ;) So cut off messages are not the fault of sockpp.  In the "Console" tab of CDT i see an error

Connection is closed, can't dispatch pending call

I suspect that V8 simply finished execution, in which case I stop the WebSocket server and dispose of the inspector.

To be precise, I want to debug a function of a WebAssembly module, called "run".  This is how I invoke the function through the V8 API:

/* Get the exports of the created WebAssembly instance. */

auto exports = instance->Get(context, V8STR("exports")).ToLocalChecked().As<v8::Object>();

auto run = exports->Get(context, V8STR("run")).ToLocalChecked().As<v8::Function>();

/* Invoke the exported function `run` of the module. */

args_t args { v8::Int32::New(isolate_, wasm_context.id), };

const uint32_t head_of_heap = run->Call(context, context->Global(), 1, args).ToLocalChecked().As<v8::Int32>()->Value();

Now how can I break/pause on entry to that function and inspect it in CDT? I tried invoking 

v8_inspector_session->schedulePauseOnNextStatement(...);

But that did not help...

I am confused and lost :/

Your help is very appreciated Elmi

[Elmi Ahmadov]

There could a few things that influence cut off messages, could you please post how do you convert V8 `StringBuffer` message to a string format that Boost can send to CDT.

you have to implement `runMessageLoopOnPause` and `schedulePauseOnNextStatement` methods to stop the execution.

If you don't implement your waiting logic in `runMessageLoopOnPause` it will just execute without waiting a user action (I changed my example to see how it behaves).

If you have an example WebAssembly source code maybe you can use my example to see how it works.

Analyzing the WebSocket messages really helps. I remember that I spend almost one or two weeks to understand how it works.

[Elmi Ahmadov]

Hi Immanuel,

Do you have any progress? Did you solve "Connection is closed" issue?

[Immanuel Haffner]

Hi Elmi,

sorry for being quiet for two weeks. I was on vacation. I will investigate your latest remarks regarding string conversion and the message loop. I will reply here. Would you be available for a call? Via Discord, Zoom, or the like...

Regards

[Immanuel Haffner]

I somehow thought that the WebSocket connections needs to be in a separate thread, so the communication with CDT can happen concurrently.  After reinspecting your code I found that this is not the case. (I don't know why I had that idea.) The threading was causing my quite some headache. The execution of my Wasm would proceed without waiting for the inspector. Anyways, now I'm having problems how to debug my Wasm code.

Without the inspector, I'd execute my Wasm code as follows:

    /* Invoke the exported function `run` of the module. */

    args_t args { v8::Int32::New(isolate_, wasm_context.id), };

    const uint32_t head_of_heap =

        run->Call(context, context->Global(), 1, args).ToLocalChecked().As<v8::Int32>()->Value();

However, with the inspector, I don't know how to "schedule" the call to `run` for debugging. Do I have to invoke this function from the CDT console? I tried to register `run` in `Global` to call it, but I'm failing to do so. Further, I cannot see the source of the Wasm module in CDT. Any clues?

Regards

[Immanuel Haffner]

I think I am close! The embedded V8 is connected to CDT. However, I can't see the executed Wasm code. I use Binaryen to generate Wasm code and write its binary representation into a byte array. Then I create a new `ArrayBuffer` from that byte array and pass it to the `WebAssembly.Module` constructor to create a Wasm module object. Last, I call `WebAssembly.Instance(module, imports)` to instantiate the just created Wasm module with the given imports.  Then I call one of the exported functions of the module instance.  This is the point where I want to start debugging!  However, calling into the function directly executes it entirely. (Setting break points didn't help.) I also tried making the function globally accessible so I could call it from the JS shell, but no success in doing so.

I saw from your example that you implemented `V8InspectorListener` to provide the source map to CDT. How can I provide the source of my Wasm code to CDT? There is no source file and no source map. There is only the in-memory representation of Binaryen and the binary format in the byte array. I could, however, emit the textual representation of the Wasm code. No clue how to provide that to CDT, though.

Please help :)

[Elmi Ahmadov]

Hi Immanuel,

Unfortunately, I've never worked with WebAssembly until today :) and I conducted a simple experiment and I improved my example to support WebAssembly (but very naively), you can see my changes here: https://github.com/ahmadov/v8_inspector_example/tree/wasm

I have successfully established a debugger session:

and I can also see the compiled WASM code in CDT (you can see in the left side under `wasm`):

Also, I can set a breakpoint in the WASM code!

I used this `emcc` tool (https://emscripten.org/docs/tools_reference/emcc.html) to compile C source code to WASM binary file.
I added a simple C source code in `example` folder and I compiled the source code via this following command:

emcc --no-entry -Os -s add.c -o add.wasm

I also tried to compile the WASM code with `-g` flag which should include source-map to the original C source file, I saw the file name add.c in CDT but I didn't see the content.

If you want to see the result yourself, first compile add.c to generate add.wasm then run it. Hope this will help.

Best regards.

[Immanuel Haffner]

Thanks a lot for providing the example. You are creating a `v8::Script` instance of the code that fetches and executes the WebAssembly stored in a wasm file. In my setting, I cannot do that. I have only the in-memory binary representation of the compiled Wasm. I don't see how to create a `v8::Script` for that such that I could invoke `Run()`. Maybe this is why I can't see the Wasm in CDT...

[Elmi Ahmadov]

Do you need to execute `Run()` on JS side or C++ side and in which side do you need the result of `Run()`? You can add a global method in JS side to fetch WASM binary from memory and you can execute on JS side as follows:

// in js
const wasmBinary = getWASMFromMemory(); // -> this will be a new global method to transfer WASM binary to JS side which you need to implement in V8 C++ side.
const module = await WebAssembly.compile(wamBinary);
const instance = await WebAssembly.instantiate(module, importsObject);
instance.exports.Run();


Maybe I got wrong idea what you want.

[Immanuel Haffner]

Hi Elmi,

I found the biggest problem yet: I need to create a `v8::Script` and invoke `run()` so that it appears in the CDT debugger. To do so, I create a temporary JS file with (roughly) the code you provided and the Wasm binary code inside the JS in form of a Uint8Array. I can now see the JS code that instantiates the module and invokes `run()`.

However, when I set a breakpoint at `run()` and then step inside, I don't see any code. I see the scope and heap. But I'd really like to see the Wasm (in Wast form).

I think I did a bad job explaining what I'm after. I am receiving a program in some graph form. To execute the program, I directly compile it to Wasm via Binaryen. Then I want to execute it in embedded V8. Because my graph to Wasm compiler is WIP, I need a debugger to investigate some bugs. But there is more to that. I patched V8 such that I can replace the memory of a Wasm instance. This allows the embedder (C++ code) and Wasm module to operate on the same memory. No more need to copy data back and forth. Basically, the embedder instantiates a Wasm module through the V8 API. Then, with my patched in function, the embedder provides a pointer to a memory location as well as the size in bytes and this region is then used as the module instance's linear memory.

I will also have to see how I can "patch" the module instance's memory in the approach I now use for debugging. Since it's a JS script, the patching of the instance's memory does not happen anymore. Maybe I can somehow hook the `WebAssembly.Instance` constructor or provide a JS function that calls back into the embedder to perform the patching? We will see...

Thanks a lot!

Kind regards

[Immanuel Haffner]

Turns out the Wasm code not showing up is a known CDT bug. After updating and restarting chromium I see the Wasm code in the CDT debugger and can set breakpoints. Brilliant!!!

Thanks again very much for your great help. I really appreciate it.

[Elmi Ahmadov]

Hi Immanuel,

It's really good to hear that!! It's my pleasure.

Best regards.