Trying to run ffmpeg.wasm in chrome extension MV3 using sandbox and web worker

[piyush gupta]

I'm trying to run ffmpeg.wasm in an MV3 extension. 

Because remote code execution seems to be allowed only in sandbox, I created a sandbox page and ran the following code provided in the above library's example.

// import { FFmpeg } from '@ffmpeg/ffmpeg';
// import { fetchFile } from '@ffmpeg/util';
function() {
    const [loaded, setLoaded] = useState(false);
    const ffmpegRef = useRef(new FFmpeg());
    const videoRef = useRef(null);
    const messageRef = useRef(null);

    const load = async () => {
        const baseURL = 'https://unpkg.com/@ffmpeg/co...@0.12.2/dist/umd'
        const ffmpeg = ffmpegRef.current;
        ffmpeg.on("log", ({ message }) => {
            messageRef.current.innerHTML = message;
        });
        // toBlobURL is used to bypass CORS issue, urls with the same
        // domain can be used directly.
        await ffmpeg.load({
            coreURL: await toBlobURL(
                `${baseURL}/ffmpeg-core.js`,
                "text/javascript",
            ),
            wasmURL: await toBlobURL(
                `${baseURL}/ffmpeg-core.wasm`,
                "application/wasm",
            ),
        });
        setLoaded(true);
    }

    const transcode = async () => {
        const ffmpeg = ffmpegRef.current;
        await ffmpeg.writeFile(
            "input.avi",
            await fetchFile('/video/video-15s.avi')
        );
        await ffmpeg.exec(['-i', 'input.avi', 'output.mp4']);
        const data = await ffmpeg.readFile('output.mp4');
        videoRef.current.src =
            URL.createObjectURL(new Blob([data.buffer], {type: 'video/mp4'}));
    }

    return (loaded
        ? (
            <>
                <video ref={videoRef} controls></video><br/>
                <button onClick={transcode}>Transcode avi to mp4</button>
                <p ref={messageRef}></p>
            </>
        )
        : (
            <button onClick={load}>Load ffmpeg-core</button>
        )
    );
}

Here is the link to example code on ffmpeg-wasm website: Usage | ffmpeg.wasm (ffmpegwasm.netlify.app)

This adds a button `Load ffmpeg-core` on page,  upon clicking that it loads remote code for ffmpeg-core and then creates a web worker, which then can be used to perform operations on media.

CSP in manifest looks like this:

content_security_policy: {
"sandbox": "sandbox allow-scripts allow-forms allow-popups allow-modals; script-src 'self' 'unsafe-inline' 'unsafe-eval'; script-src-elem 'unsafe-eval' 'unsafe-inline' 'self' https://*; object-src 'self'; child-src 'self'; worker-src 'self' blob: ;"
}

The problem is when I try to run it in sandbox page, it throws this error: 

classes.js:104  Uncaught (in promise) DOMException: Failed to construct 'Worker': Script at 'chrome-extension://xxxxx/node_modules_pnpm_ffmpeg_ffmpeg_0_12_4_node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.js' cannot be accessed from origin 'null'.
    at FFmpeg.load (chrome-extension://xxxxx/ffmpegPage.js:2243:28)

Any help is appreciated.

[wOxxOm]

Judging by the error you want to load <...>worker_js.js from your extension directory from the sandbox page. Since it doesn't belong to the security origin of the extension, you'll need to add this script to web_accessible_resources. Alternatively, you can probably put that code inside a string and convert it to a data: or blob: URL, then use it to create the worker.

[piyush gupta]

Sorry I missed mentioning that, but it was already added like this:

"web_accessible_resources": [

    {

      "resources": [

        "assets/**",

        "node_modules_pnpm_ffmpeg_ffmpeg_0_12_4_node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.js"

      ],

      "matches": [

        "<all_urls>"

      ]

    }

  ],

Also about putting the code in string, not sure how to do that because its a ffmpeg.wasm library file which is built by webpack.

Thoughts ?

[wOxxOm]

1. Maybe you didn't reload the extension after adding web_accessible_resources? Otherwise I'd consider this is a bug in the implementation of sandboxed extension pages, which you can report in https://crbug.com, assuming it wasn't already reported.

2. ** is a node glob pattern but extensions use simple wildcards, so it should be just *

3. Not WASM but the code of this js file. Something like this: new Worker(URL.createObjectURL(new Blob(['your code here'])))

[piyush gupta]

1. reloaded multiple times, and sure I can go thr crbug route but would want to exhaust any help I can get here.
2. didn't know that, updated, didn't change the error
3. ohh I didn't mean the wasm file, I meant the js library file only, let me put more context for this file
   1. file content after webpack build:  ffmpeg-worker.js (github.com)
   2. file content before webpack :  ffmpeg.wasm/packages/ffmpeg/src/worker.ts at main · ffmpegwasm/ffmpeg.wasm (github.com)
   3. place in ffmpeg library where this worker flie is used:  ffmpeg.wasm/packages/ffmpeg/src/classes.ts at main · ffmpegwasm/ffmpeg.wasm (github.com)
   4. if you suggest some changes in the library above, can try that as well with a local monkey patch

[wOxxOm]

Yeah, I guess you'll have to monkey-patch the library and apparently it's a bug in Chrome indeed because sandboxed pages should be able to load any js files from extension directory regardless of web_accessible_resources. Another guess: maybe the files should be inside the directory of the sandboxed html or in a sub-directory, but not outside of it?

[piyush gupta]

Tried patching the library with the blob url suggestion.

The original code for reference: https://github.com/ffmpegwasm/ffmpeg.wasm/blob/main/packages/ffmpeg/src/classes.ts#L166

The worker.js being referenced below: https://github.com/ffmpegwasm/ffmpeg.wasm/blob/main/packages/ffmpeg/src/worker.ts

Original code : 

this.#worker = new Worker(new URL("./worker.js", import.meta.url), {

                type: "module",

            });

patched code:

const workerUrl = new URL("./worker.js", import.meta.url);

const workerBlob = new Blob([

importScripts(' + JSON.stringify(workerUrl) + ')',

{type: 'application/javascript'});

const blobUrl = window.URL.createObjectURL(workerBlob);

this.#worker = new Worker(blobUrl, {

    type: "module",

});

Now this is the error I get: Refused to cross-origin redirects of the top-level worker script.

[wOxxOm]

My suggestion was to embed the actual contents of worker.js, which arguably should be done after build i.e. in a separate node script or as a post-compilation hook for webpack.

[Piyush]

Ahh ok I can try that as well.

But what I have done, isn't it equivalent considering its just running importscripts ?

 

And the error I got is different from what I was getting previously, so I thought it has progressed.

Also would you able to take a look in detail if I put up a git repo with minimal example ?

[wOxxOm]

1. Not the same if the bug forbids reading js files

2. Yes, try minimizing it, sometimes this process helps solve the problem, and someone besides me could look into it as well.

[piyush gupta]

Here's the sample git repo:  piyu-sh/ffmpeg-worker-extension (github.com)

Interestingly in this repo I think the worker is being created(might be wrong) because it now fails while executing the code within worker.

These are the errors I get now:

node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js:90  Refused to load the script 'blob:chrome-extension://cnmcldjfdnppmcbeninnephgplfijljf/ee688f83-8990-4383-8555-503cf8439217' because it violates the following Content Security Policy directive: "script-src 'self' 'wasm-unsafe-eval' 'inline-speculation-rules' http://localhost:* http://127.0.0.1:*". Note that 'script-src-elem' was not explicitly set, so 'script-src' is used as a fallback.

load @ node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js:90
self.onmessage @ node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js:160
node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js:90  Refused to load the script 'blob:chrome-extension://cnmcldjfdnppmcbeninnephgplfijljf/ee688f83-8990-4383-8555-503cf8439217' because it violates the following Content Security Policy directive: "script-src 'self'". Note that 'script-src-elem' was not explicitly set, so 'script-src' is used as a fallback.

load @ node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js:90
self.onmessage @ node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js:160
Sandbox.jsx:27  Uncaught (in promise) Error: Cannot find module 'blob:chrome-extension://cnmcldjfdnppmcbeninnephgplfijljf/ee688f83-8990-4383-8555-503cf8439217'
load @ Sandbox.jsx:27

Although I do have script-src-elem in manifest, but still it throws these errors.

Please check the sample git repo I posted earlier in this msg.

[wOxxOm]

Your build doesn't translate manifest.json's "assets/html/ffmpeg.html" to the output path "ffmpeg.html" so the iframe is not sandboxed so it doesn't use the CSP you specified in manifest.json. I don't see how you embed the contents of node_modules_ffmpeg_ffmpeg_dist_esm_worker_js.bundle.js into a literal string to circumvent the bug in Chrome at this.#worker = new Worker(...

[piyush gupta]

Ahh thanks for that manifest one, fixed that.

About the embedding contents in the file, couldn't really think of any direct way to do it because it's a webpack generated file, can't really put it in a string literal.

But this is what I tried:

Original worker load src:

this.#worker = new Worker(new URL("./worker.js", import.meta.url), {

                type: "module",

            });

Updated code:

const res = await fetch(new URL("./worker.js", import.meta.url))

const content = await res.text()

const blob = new Blob([content], { type: "text/javascript" })

this.#worker = new Worker(URL.createObjectURL(blob), {

    type: "module",

});

And when I put some breakpoints, it did create a blob with the contents of the script.

But this is the error I get now, which I was getting in my original code:

Refused to cross-origin redirects of the top-level worker script.

Also you can check the recent commits in the example repo: piyu-sh/ffmpeg-worker-extension (github.com)

[wOxxOm]

Well, try debugging it and find the code that performs redirection.

[piyush gupta]

Tried but can't really figure out what's causing this error

Refused to cross-origin redirects of the top-level worker script.

For reference, this is the worker script, if anyone might be able to help at some point.

[PhistucK]

Look at the network panel, do you see any redirect-related status code from unpkg.com (which the script seems to importScript from)?

If so, maybe it is just a matter of specifying a specific version somewhere.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/aea6d407-b187-49b7-a671-c5fd5bac86een%40chromium.org.