Need help migrating local storage between webview implementations

[Norman Breau]

Hi there, I'm not sure if this is the best place to ask this but I'll explain.

Long story short, I'm trying to migrate an android cordova app that is currently using the deprecated crosswalk webview (which uses chrome 53) to use the android system webview. The gotcha is, I absolutely need to maintain the data stored in local storage.

Here's what I've done so far:

- I've learned that in crosswalk/chrome 53, local storage data is stored in "/data/user/0/<app-id>/app_xwalkcore/Default/Local Storage/" which is where an SQLite database exists.

- I've also learned that at some point, chrome has switched from using SQLite to LevelDB. The LevelDB on chrome 81 is stored at "/data/user/0/<app-id>/app_webview/Default/Local Storage/"

- Using a cordova plugin, I using the android's sqlite api to read the xwalk's SQLite database. Using a third-party LevelDB api, I write the data to the levelDB database, using the proper key/value formats as I saw in the chromium source, and through careful inspection of the binary data.

All of above works, when the phone is using an android webview running chrome 81. However, when my migration process on Android 5.0/Webview 77, it failed because the levelDB database is stored in a different location.

So I have a few of questions:

1) At approximately what version did Chromium switch between using SQLite to LevelDB for local storage? Is SQLite for local storage so ancient that anybody running Android 5.0 won't being using a chrome webview that uses SQLite to store local storage data?
2) Is there any java API or a reliable way to determine where the webview will store it's leveldb database? I've attempted to just check for different file locations to see which paths exists; however during the cordova init process where the plugin runs, the entire app_webview directory doesn't appear to be created yet.

3) I am not familiar with the chromium codebase at all nor am I very well experienced in C/C++, but during my research I stumbled upon a migrateData function at https://source.chromium.org/chromium/chromium/src/+/master:components/services/storage/dom_storage/local_storage_impl.cc;l=339?q=local_storage_impl&ss=chromium&originalUrl=https:%2F%2Fcs.chromium.org%2F 

If I place a SQLite database in a certain directory, would chrome automatically migrate the local storage data for me?

If any other approaches comes to mind, I would gladly like to hear them.

Thanks in advance for any assistance you can provide.

[Marijn Kruisselbrink]

On Wed, May 13, 2020 at 11:30 PM Norman Breau <normanb...@gmail.com> wrote:

Hi there, I'm not sure if this is the best place to ask this but I'll explain.

Long story short, I'm trying to migrate an android cordova app that is currently using the deprecated crosswalk webview (which uses chrome 53) to use the android system webview. The gotcha is, I absolutely need to maintain the data stored in local storage.

Here's what I've done so far:

- I've learned that in crosswalk/chrome 53, local storage data is stored in "/data/user/0/<app-id>/app_xwalkcore/Default/Local Storage/" which is where an SQLite database exists.

- I've also learned that at some point, chrome has switched from using SQLite to LevelDB. The LevelDB on chrome 81 is stored at "/data/user/0/<app-id>/app_webview/Default/Local Storage/"

- Using a cordova plugin, I using the android's sqlite api to read the xwalk's SQLite database. Using a third-party LevelDB api, I write the data to the levelDB database, using the proper key/value formats as I saw in the chromium source, and through careful inspection of the binary data.

All of above works, when the phone is using an android webview running chrome 81. However, when my migration process on Android 5.0/Webview 77, it failed because the levelDB database is stored in a different location.

So I have a few of questions:

1) At approximately what version did Chromium switch between using SQLite to LevelDB for local storage? Is SQLite for local storage so ancient that anybody running Android 5.0 won't being using a chrome webview that uses SQLite to store local storage data?

I think the switch happened with chrome 61, so almost 3 years ago.

2) Is there any java API or a reliable way to determine where the webview will store it's leveldb database? I've attempted to just check for different file locations to see which paths exists; however during the cordova init process where the plugin runs, the entire app_webview directory doesn't appear to be created yet.

Sorry, not sure. I think the location only changed once (with chrome 79), so the two locations you've found should be the only two that matter at the moment. Of course I can't predict the future.

3) I am not familiar with the chromium codebase at all nor am I very well experienced in C/C++, but during my research I stumbled upon a migrateData function at https://source.chromium.org/chromium/chromium/src/+/master:components/services/storage/dom_storage/local_storage_impl.cc;l=339?q=local_storage_impl&ss=chromium&originalUrl=https:%2F%2Fcs.chromium.org%2F 

If I place a SQLite database in a certain directory, would chrome automatically migrate the local storage data for me?

Sort of, yes... currently chrome will still look for a sqlite database if it can't find data for an origin in leveldb and migrate that data (and then delete the sqlite database). But since it's been 3 years since we switched from sqlite to leveldb we'll probably remove that migration code soon. It's not very likely that there is still data around that we would care about migrating.

If any other approaches comes to mind, I would gladly like to hear them.

Thanks in advance for any assistance you can provide.

--
--
Chromium Discussion mailing list: chromium...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Norman Breau]

Thank you. I've got my migration working by copying the sqlite files over to the directory where Chrome expects them to be, which I determine by checking if the installed webview is >= 79. It's good to know that it's possible that Chrome's migration code may be moved soon, this means it's rather important to get my update as soon as possible then. Thanks for your reply.

[Torne (Richard Coles)]

On Thu, 14 May 2020 at 13:12, Norman Breau <normanb...@gmail.com> wrote:

Thank you. I've got my migration working by copying the sqlite files over to the directory where Chrome expects them to be, which I determine by checking if the installed webview is >= 79. It's good to know that it's possible that Chrome's migration code may be moved soon, this means it's rather important to get my update as soon as possible then. Thanks for your reply.

Yes, migrating this way is the best option - trying to read/write Chrome's internal data formats directly is likely to have issues and weird edge cases. We generally don't want apps to make any assumptions at all about the contents of the WebView data directory: neither the layout/locations nor the data formats, which is why there are no APIs for this. Since your app is only going to have to do this once for this migration, though, encoding the local storage location for this purpose seems reasonable.

 

On Thursday, May 14, 2020 at 3:30:14 AM UTC-3, Norman Breau wrote:

Hi there, I'm not sure if this is the best place to ask this but I'll explain.

Long story short, I'm trying to migrate an android cordova app that is currently using the deprecated crosswalk webview (which uses chrome 53) to use the android system webview. The gotcha is, I absolutely need to maintain the data stored in local storage.

Here's what I've done so far:

- I've learned that in crosswalk/chrome 53, local storage data is stored in "/data/user/0/<app-id>/app_xwalkcore/Default/Local Storage/" which is where an SQLite database exists.

- I've also learned that at some point, chrome has switched from using SQLite to LevelDB. The LevelDB on chrome 81 is stored at "/data/user/0/<app-id>/app_webview/Default/Local Storage/"

- Using a cordova plugin, I using the android's sqlite api to read the xwalk's SQLite database. Using a third-party LevelDB api, I write the data to the levelDB database, using the proper key/value formats as I saw in the chromium source, and through careful inspection of the binary data.

All of above works, when the phone is using an android webview running chrome 81. However, when my migration process on Android 5.0/Webview 77, it failed because the levelDB database is stored in a different location.

So I have a few of questions:

1) At approximately what version did Chromium switch between using SQLite to LevelDB for local storage? Is SQLite for local storage so ancient that anybody running Android 5.0 won't being using a chrome webview that uses SQLite to store local storage data?
2) Is there any java API or a reliable way to determine where the webview will store it's leveldb database? I've attempted to just check for different file locations to see which paths exists; however during the cordova init process where the plugin runs, the entire app_webview directory doesn't appear to be created yet.

3) I am not familiar with the chromium codebase at all nor am I very well experienced in C/C++, but during my research I stumbled upon a migrateData function at https://source.chromium.org/chromium/chromium/src/+/master:components/services/storage/dom_storage/local_storage_impl.cc;l=339?q=local_storage_impl&ss=chromium&originalUrl=https:%2F%2Fcs.chromium.org%2F 

If I place a SQLite database in a certain directory, would chrome automatically migrate the local storage data for me?

If any other approaches comes to mind, I would gladly like to hear them.

Thanks in advance for any assistance you can provide.

--