SQLite wasm and OPFS

[Randy]

I've read through the document located here:

https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system/

It says that "Browsers will not serve Wasm files from file:// URLs" and that a web server is required, along with setting 2 response headers. What about inside a Chrome extension? Can extensions create a SQLite database using SQLite wasm and OPFS?

Thanks,

  Randy

[Jackie Han]

I ran the sample code in the article in Chrome extension and it works.

Files in Chrome extensions like https://localhost (not file://), it is a secure context.

Why does it need the "Cross-Origin-Opener-Policy" and "Cross-Origin-Embedder-Policy" HTTP headers? Because it is the security requirements of SharedArrayBuffer.

As MDN's sample code, in my test (Chrome 108 stable), SharedArrayBuffer works in extensions without those two HTTP headers.

// in an extension page

const myWorker = new Worker('worker.js');

const buffer = new SharedArrayBuffer(16);
myWorker.postMessage(buffer);

In devtools' network panel, you can see those two http headers and above code works without throwing errors.

Furthermore, Chrome extension does support these two HTTP headers. You can add below contents in extension manifest.json.

// extension manifest.json

"cross_origin_embedder_policy": {
  "value": "require-corp"
},
"cross_origin_opener_policy": {
  "value": "same-origin"
},

In this page( https://developer.chrome.com/docs/extensions/mv3/manifest/ ), you can find "cross_origin_embedder_policy" and "cross_origin_opener_policy". The docs say they were introduced in Chrome 93.

After adding "cross_origin_embedder_policy" and "cross_origin_opener_policy" in manifest, you will see these two HTTP headers in devtools' network panel when you open an extension page.

In addition, Chrome supports WASM since Chrome 103.

// extension manifest.json

"content_security_policy": {
  "extension_pages": "script-src 'self' 'wasm-unsafe-eval'; object-src 'self';"
},

In summary, SQLite Wasm with SharedArrayBuffer works in Chrome extension.

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/99051174-8085-412f-8baf-51b9290178adn%40chromium.org.

[Jackie Han]

Fix typo, "In devtools' network panel, you can see those two http headers" -> "there are no these two headers and it works without error" (in the first example).

[Randy]

Thanks for the detailed response.

[Semih]

Hi Jackie,

I also tried running the demo in a Chrome extension, but it fails because it says OPFS is not available. Digging into it, I saw that it checks a bunch of things shown below to determine OPFS support and fails because createSyncAccessHandle is not available.

if(!self.SharedArrayBuffer ||

    !self.Atomics ||

    !self.FileSystemHandle ||

    !self.FileSystemDirectoryHandle ||

    !self.FileSystemFileHandle ||

    !self.FileSystemFileHandle.prototype.createSyncAccessHandle ||

    !navigator.storage.getDirectory){

   return Promise.reject(

     new Error("This environment does not have OPFS support.")

   );

}

I tried looking around for some info, but could not see anything.

A bare minimum non-working example is here:

https://github.com/shazar/sqlite3-wasm-chrome-extension

What am I missing?

[Jackie Han]

Hi, Semih

Comparing your code with the sample code, I found several problems.

1. createSyncAccessHandle is only available in web worker (e.g. new Worker('worker.js') ) , not in service worker. Your code runs it in service worker.

2. Your code only includes sqlite3.js and sqlite3.wasm. But I see the sample code includes more files than that.

I use that sample code and run as a web worker, it works (sqlite3.opfs is available).

[Semih]

Thanks for the reply.

I thought service workers were just web workers in extensions.

But it seems like I cannot create a Worker from a service worker as it says Worker is not available.

Also tried creating one from the content script, but says a Worker cannot be created. I've updated the repo with the new test code and the README with the error, you can see here:

https://github.com/shazar/sqlite3-wasm-chrome-extension

PS: I also added the missing sqlite files, but I don't think the problem I'm having now is related to them.

[Jackie Han]

Service worker and web worker are different. You can read their introductory article. Here, createSyncAccessHandle is a sync IO method, so it only works in web worker by design. You can create a web worker in extension pages, e.g. the popup page or any web pages in your extension.

Content scripts are out of extension origin. You should only use sqlite in your extension origin.

You can't create web worker in service worker at present. Some extension developers propose that web workers can be created in service worker.

https://github.com/w3c/webextensions/issues/287

https://bugs.chromium.org/p/chromium/issues/detail?id=1219164

[Semih]

OK, that clears it up. Thank you!

[Randy]

One thing I have discovered is that you can't use the OPFS Explorer (which is a Chrome extension) to debug the OPFS of a Chrome extension. That's disappointing. It would be much more helpful if the OPFS Explorer was like all the other Storage sections (Local Storage/IndexedDB/etc) of the Application tab in Chrome DevTools.

[Cuyler Stuwe]

I think long before we get any sort of OPFS visibility, it would be nice just to have e.g., `chrome.storage.*` visibility for starters.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/d67eb0e2-5a42-4481-aaab-c10122289fb6n%40chromium.org.

[Vince Scafaria]

Hi, this looks like just the right discussion thread to ask the question...

Is there a "hello world" example of a Chrome Extension with SQLite for OPFS. I see on the thread that Jackie got it to work, but I'm somewhat new to this. 

I have a manifest with:

{
  "manifest_version": 3,
  "name": "Test App",
  "version": "1.0.0",
  "description": "Testing...",
  "icons": {
    "16": "images/icon-16.png",
    "32": "images/icon-32.png",
    "48": "images/icon-48.png",
    "128": "images/icon-128.png"
  },
  "action": {
    "default_title": "Run Sqlite test",
    "default_icon": {
      "16": "images/icon-16.png",
      "32": "images/icon-32.png",
      "48": "images/icon-48.png",
      "128": "images/icon-128.png"
    }
  },
  "content_scripts": [
    {
      "js": ["content.js"],
      "matches": ["*://*/*"]
    }
  ],
  "permissions": [
    "activeTab"
  ],

    "cross_origin_embedder_policy": {
    "value": "require-corp"
  },
  "cross_origin_opener_policy": {
    "value": "same-origin"
  }
}

and a content.js with

const myWorker = new Worker('worker.js');
const buffer = new SharedArrayBuffer(16);
myWorker.postMessage(buffer);

and I get 

Uncaught ReferenceError: SharedArrayBuffer is not defined

Thank you.

[Randy]

If it helps, I took the example code from the article mentioned at the start of this thread and used Jackie's notes to convert it to run in an extension:

https://github.com/randyl/sqlite3-wasm-demo-extension

[Vince Scafaria]

Oh wow, that works for me. Thanks so much! 

[Vince Scafaria]

Is it possible to save to SQLite from the background.js? I see this link suggesting that it can't be done in MV3?
https://stackoverflow.com/questions/73755145/how-to-create-a-web-worker-from-an-extension-service-worker
https://bugs.chromium.org/p/chromium/issues/detail?id=1219164

Should I be calling the database from my content script file instead? (Is that even possible)?

The github sample that worked from Randy was in a standalone html page. My use case is a chrome extension that lets the user scrape html content from the page they're viewing into a database. Thanks. 

[Oliver Dunk]

Hi Vince,

As mentioned earlier, there's a missing API in service workers which means the OPFS SQLite implementation doesn't work at the moment. Thomas Steiner who has been working on this left a reply here: https://stackoverflow.com/questions/75461104/saving-to-sqlite-from-chrome-extension/75461770#75461770

In the meantime, your best option would be opening a new extension HTML page if you'd like to use this.

Offscreen documents (https://developer.chrome.com/docs/extensions/reference/offscreen/) may also be an option but I'm unsure if we'd allow it in any of the current reasons.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/6e42cce7-de13-4d3c-8552-62d27e16fdfen%40chromium.org.

[Vince Scafaria]

Thank you. Re "I'm unsure if we'd allow it in any of the current reasons", I see from that page documentation that it's severely restricted. Taking a step back, my use case boils down to:
(a) User clicks extension button to launch popup to say "record" (recording certain types of elements on the page). 
(b) Current plan was to periodically persist scraped elements to SQLite. Perhaps instead I persist them to local storage and then when the page is fully analyzed (allowing for scrolling),

(c) *either* (i) (preferred if works smoothly) launch an offscreen page so that local storage persists to SQLite invisibly [is there a best example of this?] (ii) or else I'd launch a page saying "done scraping! ... now saving to local database..." using a dedicated tab like the example here (https://github.com/randyl/sqlite3-wasm-demo-extension). 

Any additional thoughts on my use case would be much appreciated, thank you! 

[Oliver Dunk]

Hi Vince,

I just wanted to check in and say that I'm still discussing this with the team. None of our current reasons seem appropriate, so I'm trying to learn more about OPFS in extensions and if we might want to add a new reason for example.

I'll try to loop back again soon.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Oliver Dunk]

Hi again,

We've just added a new WORKERS reason which can be used to spawn a worker from an offscreen document: https://chromium.googlesource.com/chromium/src.git/+/a8fe947f9d5737f4e36fe32ab8fa851ea6da8865. This is currently in Canary and will be shipping to stable in Chrome 113 (currently targeted for May 2nd).

While I'd still ultimately love to see https://crbug.com/1219164 fixed, this provides an (albeit multi-step) way of opening an offscreen document and then creating a worker from inside of it. I'm hopeful it at least makes sure we enable as many use cases as possible in the short term.

We'll be updating our documentation soon, but there is currently no lifetime enforcement, i.e the offscreen document opened for this purpose can live forever. Please do try to avoid abusing that though and write things in a way where you clean up when you're done!

Vince - hopefully this gives you what you are looking for.

Thanks all for the feedback. It directly contributed to adding the new reason :)

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Vince Scafaria]

This is very exciting! Thank you for prioritizing it. 

[Vince Scafaria]

Hello! Sharing a link from this thread to a new thread sharing code for SQLite with OPFS (hopefully toward the above-mentioned background workers!).

https://groups.google.com/a/chromium.org/g/chromium-extensions/c/cSCYcM9oJsk 

"Our non-profit created an open source project "Whosum Social Assistant" that uses Chrome's awesome SQLite (OPFS) in-browser database (thank you!). I welcome any thoughts, feedback, or collaboration"

[Vince Scafaria]

Hi Oliver, I am excited to see Chrome 113 is released and am ready to implement "the new WORKERS reason" for SQLite via OPFS in the background. Can you please help me understand how to use it for my use case above "launch an offscreen page so that local storage persists to SQLite invisibly [is there a best example of this?]". My current code is open source here where I currently don't/can't access the database until the user launches the dedicated index.html page. The worker is currently launched from my index.js code (similar to to Randy's simple example) without specifying any "reason" so I'm not quite clear on how to make use of a new reason being available. 

Ideally I would like to be able to read/write to the database (a) related to the content the user is browsing and also (b) content obtained via background "fetch" commands. Sample use case for (a): Apply html highlighting to posts from your "favorite" people (favorites stored in the database) while browsing a social media site; and add more favorites to the database while browsing. Sample use case for (b): Request related information from a web server (e.g. reputation score etc.) and persist it to the database either while browsing or while reviewing the database UI. Thanks! 

[Oliver Dunk]

Hi Vince,

I haven't had a chance to look over all of your source code, but the flow would be something like this:

1. In your content script, send a message to the background, e.g chrome.runtime.sendMessage({ name: "set-data", value: "hello-world" });
2. In your background page, listen to this message via. chrome.runtime.onMessage
3. If you haven't already, create an offscreen document, following the example here and using the WORKERS reason: https://developer.chrome.com/docs/extensions/reference/offscreen/#example
4. In your offscreen document, call new Worker().
5. Post a message to the worker.
6. In the worker, listen for the message and write to OPFS.

Hope that helps!

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Vince Scafaria]

Interesting. I'll try that shortly! A question  related to these offscreen documents (which I haven't yet used) : Am I able to spawn an offscreen document for a domain different from the one currently displayed to the user, and scrape that page for the user? Wondering about CORS etc. In an ideal world (not sure if this is possible), I'd like for my app to help the user collect related information about the person they're viewing (e.g. their other social media posts) and database it and/or splice that into the html they're viewing. 

Use case: A user has indicated who their favorite social media people are (and databased those favorites), and now they'd like to see a unified social stream drawing content for those individuals. The logged-in user has permission to view that social data, but those other social sites (e.g. YouTube, Twitter) are not the active-focus site. 

Thanks!

[Oliver Dunk]

Hi Vince,

Offscreen documents themselves have to be local files within your extension - but you could spawn one with the DOM_SCRAPING reason and embed an iframe inside of it. This iframe would get content scripts injected in to it like normal which should allow you to silently load a site and scrape information from it.

Hopefully that makes sense.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Vince Scafaria]

The DOM_SCRAPING reason sounds very promising. Thank you, I'll research these further and let you know if I have follow-up questions. 

[Vince Scafaria]

Hi Oliver, I have a follow-up question re DOM_SCRAPING and posted it here fyi (since I'm getting a bit off topic for this SQLite-related thread).  Thanks!