Intent to Deprecate and Remove: FTS extension support in WebSQL

[Victor Costan]

Primary eng (and PM) emails

pwn...@chromium.org

Summary

WebSQL was implemented in WebKit and Blink as a thin layer on top of SQLite. WebSQL has failed to get traction with other browsers, and the work on the standard has been discontinued. Due to the SQLite implementation, WebSQL exposes a full text search extension (FTS3) which has proven difficult to maintain.

Motivation

First, WebKit has removed FTS support from its WebSQL implementation, and we would like to do the same, for interoperability. Second, FTS had a security issue, and maintaining it has been troublesome. Removing it would allow us to remove a large surface area (SQLite virtual tables) from WebSQL. Third, FTS has low (apparently zero) usage. Given that WebSQL has been abandoned, we’d rather not maintain the unused parts of it.

Interoperability and Compatibility Risk

Describe the degree of interoperability and compatibility risk. For a feature that is also supported in some other engine, do they support eventual removal?

Edge: Never shipped WebSQL

Firefox: Never shipped WebSQL

Safari: Removed FTS support from WebSQL (shipped in Safari 10.1.1)

Alternative implementation suggestion for web developers

Web developers will have to implement full text search at the application level, or settle for a subset of the functionality. Gmail’s offline Chrome application uses “LIKE %needle%” queries.

Usage information from UseCounter

We have shipped a use counter for FTS use in WebSQL in M60: https://www.chromestatus.com/metrics/feature/timeline/popularity/1980

The counter is completely to zero (even looking internally at beta-only data). I think I triggered it when I checked if it works, a few days ago. I checked that the counter triggers by going to chrome://histograms after running an FTS WebSQL test like https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/storage/websql/fts-crash-703507.html

OWP launch tracking bug

https://crbug.com/742518

Entry on the feature dashboard

https://www.chromestatus.com/feature/5669618848890880

Requesting approval to remove too?

Yes. I would like to remove the feature immediately. I intend to request a merge to M60 for the removal, due to an active security issue. Given the zero UseCounter in beta, I believe no deprecation period is warranted.

[Rick Byers]

LGTM1 to remove without a deprecation period.

I double-checked the UseCounter data internally and I agree it's flat 0.

This already shipping in Safari is an additional good signal that it's web compatible (assuming you don't find people complaining when you google the error message).

I know the M60 merge bar is very high at this point, so if the merge is approved for security reasons then merging back to M60 sounds good to me despite the fact that we normally would want to verify UseCounter data from stable and get some beta feedback.  Active security issues trump web compat risk, especially when we have data like this that already suggests the risk is quite low.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP_mGKohAxsnikk1Jq0d3mEOqtyaVJJbPxJO7L4%2BQ58CWtXghw%40mail.gmail.com.

[Chris Harrelson]

LGTM2 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY_mTNV9-JWUZjrTdFQrzYokO%2BgpS-rrHtZykr%3DzuX2Jdw%40mail.gmail.com.

[TAMURA, Kent]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_BxgbM61EzMm5h9OCPSVDFDqKq%2BKWOxgeye18mXyDeaA%40mail.gmail.com.

--

TAMURA Kent
Software Engineer, Google

[con...@superhuman.com]

Hey Chrome!

As an active user/developer using FTS on WebSQL I’m surprised your counters are showing 0 (it should at least say 1

!) Are your counters accurate? (we’re using it for a very similar use-case to Offline Gmail: https://superhuman.com).

I’m strongly against this proposal for two reasons:

1. There exists no meaningful way to do offline search on the web except FTS+WebSQL.

2. It seems (to me) to be significantly less effort to fix the FTS integration than to build a world-class search engine over either WebSQL or IndexedDB.

Is there work we can usefully do to fix this problem for you?

Conrad

LGTM3

LGTM2 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY_mTNV9-JWUZjrTdFQrzYokO%2BgpS-rrHtZykr%3DzuX2Jdw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_BxgbM61EzMm5h9OCPSVDFDqKq%2BKWOxgeye18mXyDeaA%40mail.gmail.com.

[Rick Byers]

Thanks for reaching out Conrad!  It's entirely possible our UseCounter is lying to us somehow, and then that would totally change the discussion.

Can you give us (at least Victor and I) access / steps to reproduce the behavior on your site so we can dig in?  I signed up with my google.com account but it sounds like I'm on a waitlist.

Also what does your site do on non-WebKit-based browsers?  The Microsoft folks love to remind us that Outlook Web Access works just as well as GMail without relying on WebSQL at all (uses IndexedDB and libraries on top).  Without any apparent path for broad browser support, WebSQL itself is on pretty rocky footing on the web at the moment (though that's not the subject of this intent, it's certainly related).

Rick

[Conrad Irwin]

Hi Rick,

Thanks for looking into this! I've added your Google account to our private beta, let me know if you need a test account added instead (I know some large companies prevent oauth access to emails).

You can sign in here: https://mail.superhuman.com/. This will prompt you first for Oauth, and secondly to install an extension, which is what actually uses WebSQL. Happy to share our code if you need it.

Conrad

On Fri, Jul 14, 2017 at 10:14 AM, Rick Byers<rby...@chromium.org>wrote:

Thanks for reaching out Conrad!  It's entirely possible our UseCounter is lying to us somehow, and then that would totally change the discussion.

Can you give us (at least Victor and I) access / steps to reproduce the behavior on your site so we can dig in?  I signed up with my google.com account but it sounds like I'm on a waitlist.

Also what does your site do on non-WebKit-based browsers?  The Microsoft folks love to remind us that Outlook Web Access works just as well as GMail without relying on WebSQL at all (uses IndexedDB and libraries on top).  Without any apparent path for broad browser support, WebSQL itself is on pretty rocky footing on the web at the moment (though that's not the subject of this intent, it's certainly related).

Rick

[Rick Byers]

Oh yeah I definitely can't give you oath to my google.com e-mail (or install a "can access all information on all websites you visit" extension when using a chrome profile any of my real account for that matter <grin>).  I've requested access for rbyer...@gmail.com instead.  Victor, I can share the password for that account with you.

Ah if it's accessing via an extension that explains why it doesn't show up.  Generally our web compat principles / analysis are around the open web and so the data doesn't include extensions by default.  Obviously extension compatibility is important too, but it's a little different. Still it's not showing up in our extension-specific metrics so we still need to investigate that (the extensions-specific metrics haven't been tested anywhere near as thoroughly as the normal open-web metrics).

I suppose relying on a Chrome extension answers the question about other browsers, right?  This is a Chrome-only application?

On Fri, Jul 14, 2017 at 1:29 PM, Conrad Irwin <con...@superhuman.com> wrote:

Hi Rick,

Thanks for looking into this! I've added your Google account to our private beta, let me know if you need a test account added instead (I know some large companies prevent oauth access to emails).

You can sign in here: https://mail.superhuman.com/. This will prompt you first for Oauth, and secondly to install an extension, which is what actually uses WebSQL. Happy to share our code if you need it.

Conrad

On Fri, Jul 14, 2017 at 10:14 AM, Rick Byers<rby...@chromium.org>wrote:

Thanks for reaching out Conrad!  It's entirely possible our UseCounter is lying to us somehow, and then that would totally change the discussion.

Can you give us (at least Victor and I) access / steps to reproduce the behavior on your site so we can dig in?  I signed up with my google.com account but it sounds like I'm on a waitlist.

Also what does your site do on non-WebKit-based browsers?  The Microsoft folks love to remind us that Outlook Web Access works just as well as GMail without relying on WebSQL at all (uses IndexedDB and libraries on top).  Without any apparent path for broad browser support, WebSQL itself is on pretty rocky footing on the web at the moment (though that's not the subject of this intent, it's certainly related).

Rick

[Conrad Irwin]

Yes, that makes sense (risk tolerance tends to zero as employee count tends to infinity :D). I've added rbyer...@gmail.com too.

Web compatibility vs. extension compatibility makes sense, so that probably explains why we're not showing up (interestingly enough, we'd probably not be using the extension if the WebSQL API was more robust to having limited storage quota, so definitely hearing the tech creak under us).

This is chrome-only, yes. We're also working on a node-webkit version for users who don't/can't use chrome.

Conrad

On Fri, Jul 14, 2017 at 10:58 AM, Rick Byers<rby...@chromium.org>wrote:

Oh yeah I definitely can't give you oath to my google.com e-mail (or install a "can access all information on all websites you visit" extension when using a chrome profile any of my real account for that matter <grin>).  I've requested access for rbyers.test@gmail.com instead.  Victor, I can share the password for that account with you.

Ah if it's accessing via an extension that explains why it doesn't show up.  Generally our web compat principles / analysis are around the open web and so the data doesn't include extensions by default.  Obviously extension compatibility is important too, but it's a little different. Still it's not showing up in our extension-specific metrics so we still need to investigate that (the extensions-specific metrics haven't been tested anywhere near as thoroughly as the normal open-web metrics).

I suppose relying on a Chrome extension answers the question about other browsers, right?  This is a Chrome-only application?

On Fri, Jul 14, 2017 at 1:29 PM, Conrad Irwin <conrad@superhuman.com> wrote:

Hi Rick,

Thanks for looking into this! I've added your Google account to our private beta, let me know if you need a test account added instead (I know some large companies prevent oauth access to emails).

You can sign in here: https://mail.superhuman.com/. This will prompt you first for Oauth, and secondly to install an extension, which is what actually uses WebSQL. Happy to share our code if you need it.

Conrad

On Fri, Jul 14, 2017 at 10:14 AM, Rick Byers<rbyers@chromium.org>wrote:

Thanks for reaching out Conrad!  It's entirely possible our UseCounter is lying to us somehow, and then that would totally change the discussion.

Can you give us (at least Victor and I) access / steps to reproduce the behavior on your site so we can dig in?  I signed up with my google.com account but it sounds like I'm on a waitlist.

Also what does your site do on non-WebKit-based browsers?  The Microsoft folks love to remind us that Outlook Web Access works just as well as GMail without relying on WebSQL at all (uses IndexedDB and libraries on top).  Without any apparent path for broad browser support, WebSQL itself is on pretty rocky footing on the web at the moment (though that's not the subject of this intent, it's certainly related).

Rick

[Victor Costan]

On Fri, Jul 14, 2017 at 10:58 AM, Rick Byers <rby...@chromium.org> wrote:

Oh yeah I definitely can't give you oath to my google.com e-mail (or install a "can access all information on all websites you visit" extension when using a chrome profile any of my real account Ah if it's accessing via an extension that explains why it doesn't show up.  Generally our web compat principles / analysis are around the open web and so the data doesn't include extensions by default.  Obviously extension compatibility is important too, but it's a little different. Still it's not showing up in our extension-specific metrics so we still need to investigate that (the extensions-specific metrics haven't been tested anywhere near as thoroughly as the normal open-web metrics).

In case it helps with the use counters investigation, I quickly put together an extension that creates a WebSQL FTS table at https://github.com/pwnall/websql-fts-extension. I loaded this extension in Canary, and confirmed that I get an entry for feature 1980 under Blink.UseCounter.Extensions.Features when I click the extension's button.

    Victor

[Conrad Irwin]

I think the issue with this approach is that it only counts new users, almost all of our users (and presumably any extension with a database) won't trigger your counter as they already have a search table.

This is particularly a problem with browser extensions because the TTL of the database is essentially infinite.

Does installing the Superhuman extension from https://mail.superhuman.com trigger the count?

Conrad

[Rick Byers]

I've confirmed that installing the extension on Chrome 60+ triggers the count.  So the metrics seem to be working properly and our lack of data here was just due to do the combination of the metric not yet being in Chrome stable, and that it only counted new installs of Superhuman not active usage. I agree that to properly measure usage it's insufficient to measure just table creation.

An alternate fix for the security issue has now landed, so this issue is no longer time-critical.  So I rescind my LGTM.

Victor, as discussed offline, if you're still interested in pursuing FTS removal, I think the most relevant blink compat principle to focus on is Ease of adaptation.  In particular this bit: "At the extreme end, a breaking change which takes away a capability (no matter how minor) which cannot be achieved by any other mechanism is generally considered high risk. Generally we avoid breaking any use cases which cannot be shown to have a reasonable alternate implementation".  It seems like we've got a great potential partner with Superhuman who wants to move off WebSQL entirely (and relying on a Chrome extension) but feels they can't due to limitations in other storage options.  As long as they are engaged and making a serious attempt to try to move away from aspects WebSQL, sharing compelling data showing why the alternatives aren't yet good enough for them then I personally don't think we should remove pieces they're depending on. We don't make compat decisions based on any single property, but in general for every engaged developer who would be potentially impacted, there tends to be orders of magnitude more who would be affected who we don't know about. So once we know of any usage where a developer is claiming they can't reasonably accommodate the change, it's on us to prove there's a viable migration path.

Of course that's not the only relevant compat principle here.  For example, it needs to be balanced with:

• PageViews impacted - if you can fix the counter to include not just creation of such tables but access, and it makes it to stable, then we should have a much better idea of actual usage.  Given that Superhuman is in a limited beta state, I assume that in absolute terms usage will still be quite small if it's just them.
• Unique sites impacted - if we can show that virtually all of our usage is this one single product, then that makes removal more compelling (though IMHO doesn't eliminate the need to show there's some viable migration path).  We can talk offline about how you might get such data.
• Security - if FTS creates an ongoing security risk, then obviously that's additional motivation to want to remove support for it.
• Interoperability - Safari being the only other major browser with support (with no likely path for other browsers) definitely makes WebSql an attractive removal target.  Should Safari drop support for the fts3 extension, then that would make dropping support for it from Chrome more attractive as well.

Sound reasonable?  Kent, Chris, WDYT?

Thanks,

  Rick

Conrad

[Victor Costan]

Thank you for the thoughtful response, Rick!

One small clarification, below.

On Tue, Jul 18, 2017 at 5:32 AM, Rick Byers <rby...@chromium.org> wrote:

• Interoperability - Safari being the only other major browser with support (with no likely path for other browsers) definitely makes WebSql an attractive removal target.  Should Safari drop support for the fts3 extension, then that would make dropping support for it from Chrome more attractive as well.

The whole WebSQL is a more complex story, but Safari _did_ drop support for the FTS3 extension. Safari 10.1.1, the current version on my OS X 10.12 laptop, passes a layout test that checks for lack for FTS3 support. We are the only browser that ships FTS3 at the moment.

I'll collect more data and come back with an updated intent. Since the (most recent) security issue was solved, the timeline needs to be revised, at the very least.

Thank you,

    Victor

[Rick Byers]

On Tue, Jul 18, 2017 at 1:02 PM, Victor Costan <pwn...@chromium.org> wrote:

Thank you for the thoughtful response, Rick!

One small clarification, below.

On Tue, Jul 18, 2017 at 5:32 AM, Rick Byers <rby...@chromium.org> wrote:

• Interoperability - Safari being the only other major browser with support (with no likely path for other browsers) definitely makes WebSql an attractive removal target.  Should Safari drop support for the fts3 extension, then that would make dropping support for it from Chrome more attractive as well.

The whole WebSQL is a more complex story, but Safari _did_ drop support for the FTS3 extension. Safari 10.1.1, the current version on my OS X 10.12 laptop, passes a layout test that checks for lack for FTS3 support. We are the only browser that ships FTS3 at the moment.

Ah, sorry I forgot that point - thanks for correcting.  We definitely don't like shipping any feature that's Chrome-only :-)

I'll collect more data and come back with an updated intent. Since the (most recent) security issue was solved, the timeline needs to be revised, at the very least.

Sounds great.

Thank you,

    Victor

[Conrad Irwin]

On Tue, Jul 18, 2017 at 5:50 AM, Rick Byers<rby...@chromium.org>wrote:

Quick question for you Conrad: is all your WebSQL access from pages with a chrome-extension:// URL, or do you also do some from the regular web page?  I assume that if it's the extension that owns the database, you can only read from the database from the extension context, right?  I'm just trying to predict what Victor will see in his metrics when he fixes them.

Yes, that's correct.

I removed extensions from our default metrics awhile back, but I wasn't considering cases like this where a web property relies on it's own extension to function correctly.  I'm still not exactly sure what to make of web compat around extensions - it's definitely not the same thing as the open web.  In general the web platform team's mission is to make the web better, and we don't really see extensions as part of "the web", though I do see making apps like SuperHuman successful as part of our mission (just that relying on an extension is a bug we need to fix together somehow).

Sounds great to me. We're actually pretty close to not needing an extension, the main reasons we have it are:

1. When the storage quote gets full, storage acts unpredictably. Because this happens a long time before the user's hard-disk is full it's hard to communicate what's going on (and because deleting data from IndexedDB/WebSQL can cause disk allocation, we can't fix it for our users). Using an extension avoids this failure mode, as when the user's disk is full literally everything is broken, so it's fine for us to pop up an alert and tell them to delete some files from their hard disk. https://storage.spec.whatwg.org/ should help with this I think.
   
2. It gives us another process with access to a DOM implementation. A lot of our CPU time is spent parsing and processing untrusted HTML emails. Using jsdom in a web-worker to do this both reduces security (DOMPurify uses the browser's parser to avoid any vulnerabilities caused by miss-parses) and is also approximately 12 times slower (or equivalently 12 times more resource intensive). (see also https://github.com/w3c/ServiceWorker/issues/846)
   

Almost everything else (notifications, background sync) we can probably use ServiceWorkers for (though at least as they're implemented in Chrome right now, we'd have to accept some dip in quality to meet the throttles that service workers have around CPU time and background notifications).

I think OOPIFs will also be a reasonable solution for 2, assuming developer's have enough control over which frame ends up in which process, and 1. is also in progress; so I'm optimistic :)

Thanks again for reaching out so quickly on this to blink-dev.  Active security issues (especially those about to be announced publicly at a BlackHat conference) make us willing to rush decisions like this.  I'm very glad we caught your usage and found an alternate plan in time.  Sorry for the scare!

No worries, thanks for all your help!

Conrad

[harris...@acma.gov.au]

As a  web developer, FTS is a very useful part of WebSQL.   A couple of years ago, I developed an offline database web search facility in conjunction with a daily downloadable RF database.  This app (you can see it here; https://web.acma.gov.au/offline-rrl/ ) is a  valuable tool to allow our RF engineers, and public,  to lookup our RF database and search it, etc, when they are out in the field eg; “off grid” and “off network”.  WebSQL and FTS3 are key components to this tool and I do not know of any other realistic web based client side tool that can achieve the same power.  I would strongly like this functionality to remain.  And on the topic of virtual tables, any chance of extending to allow offline WebSQL spatial indexing capabilities as well as full text searching would also come in very useful.

 

Harris.

[Victor Costan]

On Thu, Aug 3, 2017 at 11:27 PM, <harris...@acma.gov.au> wrote:

As a  web developer, FTS is a very useful part of WebSQL.   A couple of years ago, I developed an offline database web search facility in conjunction with a daily downloadable RF database.  This app (you can see it here; https://web.acma.gov.au/offline-rrl/ ) is a  valuable tool to allow our RF engineers, and public,  to lookup our RF database and search it, etc, when they are out in the field eg; “off grid” and “off network”.  WebSQL and FTS3 are key components to this tool and I do not know of any other realistic web based client side tool that can achieve the same power.  I would strongly like this functionality to remain.  And on the topic of virtual tables, any chance of extending to allow offline WebSQL spatial indexing capabilities as well as full text searching would also come in very useful.

While not part of this intent, at a high level, I'm looking at ways to wind down our WebSQL support, as it's an abandoned standard. I doubt that any attempt to expand WebSQL would pass Blink's Intent to Ship process today, because it's an abandoned standard, and both Firefox and Edge have publicly opposed it.

Right now, I think that IndexedDB is the best that the Web platform has to offer for applications that have complex storage needs. If you're willing to tell us about what you're trying to build, and how the Web platform fits or doesn't fit your goal, please start a discussion thread at stora...@chromium.org -- https://groups.google.com/a/chromium.org/forum/#!forum/storage-dev

At a high level, we are aware that our IndexedDB implementation does not currently offer acceptable performance levels for some applications where WebSQL did the job well. We are working to improve that. Concrete use cases and benchmarks where IndexedDB needs to get better would be super-helpful to our efforts. If you have this information (especially benchmarks), please post on stora...@chromium.org, or e-mail me privately if you don't wish to share the information publicly.

Thanks for understanding,

     Victor

[harris...@acma.gov.au]

Hi Victor,

 

Thanks for your explanation.  The offline app I mentioned above ( https://web.acma.gov.au/offline-rrl/ ) is a complete standalone web app that can consume our daily downloadable relational database (zip file of csv’s).  It allows extensive and complex searching of this database – leveraging the browser performance of WebSQL and FTS.  I have long understood that WebSQL has been an abandoned standard, and if it really is abandoned, I for one, have a real need for a standards based native SQL engine inside the browser – preferably with full text and spatial indexing capabilities.

 

Harris