Storage questions (WAS: Re: Some q's on localStorage)

[Jeremy Orlow]

+ dumi and eric

On Thu, Feb 4, 2010 at 4:14 PM, ohle <mohle...@googlemail.com> wrote:

> > So, can a web-page (non-extension) use the sql api and consume an arbitrary
> > amount of storage (a few gig say?) "without" explicit user-approval, or is
> > there a separate quota for this?
>
> It has its own quota.  We're considering making it possible to raise the
> quota, though I don't think anything's certain yet.

Hi,

I would like to chime in on the question of the user adjustability of
the websql database size. Currently I'm evaluating the possibility to
reimplement part of a legacy POS application with means of some of the
new HTML5 features. One absolute requirement is the ability to work in
offline mode, should the internet connection not be available. For
this to work the database has to hold at least 100.000 products,
probably more, and a few supporting tables, some of which have about
the same size. The products will be synchronized in small chunks with
a low priority. No problem with a local Access or similar database but
not possible with websql as currently implemented in Chrome of Safari.

Why not?  Can't you do lots of small transactions?

Hopefully soon you'll be able to do database work on a background thread as well.

 

There are a few things which could help my particular use case:
1. Let the implementor specifiy a larger db default size on creation
of the db than currently possible. If this is greater than a certain
limit, say 5 meg, let the browser show a user dialog in which the user
gets the chance to decide whether allow the application to create a db
of the requested size or not. I know there is a certain danger that
users will tend to let the application continue without consciously
contemplating the decision but maybe a hint like "The requested
database size represents 2% of your available disk space" could help?
An absolute limit could still be implemented (maybe relative to the
available disk space) but for me it just needs to be higher than it
currently is.

I believe we're considering this for SQLDatabases.  Dumi would know more.

 

2. Give the programmer a chance to query the current db size and the
max db size and an api to request more space as needed by the
application. This way a user dialog could show up on expected
situations and not as currently implemented at any point when a db
write operation takes place.

Ditto.

 

3. Currently the only access to the locally stored data is via the
developer tools (local storage and websql). I think this would be
better placed in the browser options menu since the data is comparable
to cookies. The user should have an easy access to locally stored data
and at least be able to remove databases or control database size
(maybe in addition to 1.). The sql editor could still be left in the
developer tools.

This is planned for Chrome 5 and most of the code is already written.

 

Sad to say, but currently Google Gears or Adobe AIR would be better
options for me as both allow larger db sizes, but since Gears seems to
be phased out and AIR is incompatible with many HTML5 standards ...

Another feature from Gears I really miss the RessourceStore: I need to
make some product pictures available offline, too, and the
RessourceStore seems like a great way to achieve just that (haven't
tried it yet). I think there is an effort at the W3C with the
"programmable cache" for HTML5 to achieve something similar but there
is nothing on the Chrome roadmap, or is there?

 Eric is currently working on getting a file system API agreed upon which should satisfy this use case.

Thanks for reading

You're welcome.  Hopefully the answers are better than you expected.

[ohle]

> > I would like to chime in on the question of the user adjustability of
> > the websql database size. Currently I'm evaluating the possibility to
> > reimplement part of a legacy POS application with means of some of the
> > new HTML5 features. One absolute requirement is the ability to work in
> > offline mode, should the internet connection not be available. For
> > this to work the database has to hold at least 100.000 products,
> > probably more, and a few supporting tables, some of which have about
> > the same size. The products will be synchronized in small chunks with
> > a low priority. No problem with a local Access or similar database but
> > not possible with websql as currently implemented in Chrome of Safari.
>
> Why not?  Can't you do lots of small transactions?

The problem is not in transaction size, it is the fact that Chrome
does not allow me as a developer or any user to set a requested or
maximum database size. After about 40000 rows I get a message saying
that no more space is available for the current db (although there is
more than enough free space on the disk).

Actually I must admit that I did not do my homework right: As I found
out today Apple's Safari implements some of the ideas I posted
yesterday really nicely: As soon as a user hits a page which creates a
db of a certain size she/he is prompted with a confirmation dialog.
Should the application db size later on hit this limit, the user is
prompted again with another confirmation dialog to grow the db size
beyond the originally specified limit. In Safari's options the user
can set default max db size and inspect the current db sizes as well
as change their size limits. In addition the user can choose to delete
whole databases.

I apologize, if some of my thoughts now might seem like an
unauthorized copy of Apple's implementation but I simply did not check
thoroughly enough. But I'm still pretty sure that currently Chrome and
Opera as the other two implementors of the HTML5 Web SQL database spec
do currently not have anything comparable.

> Hopefully soon you'll be able to do database work on a background thread as
> well.

Great! This is another important requirement for my use case: I intend
to write some small chunks of data first locally and later on
synchronize the data with our central system in the background.

> > Another feature from Gears I really miss the RessourceStore: I need to
> > make some product pictures available offline, too, and the
> > RessourceStore seems like a great way to achieve just that (haven't
> > tried it yet). I think there is an effort at the W3C with the
> > "programmable cache" for HTML5 to achieve something similar but there
> > is nothing on the Chrome roadmap, or is there?
>
>  Eric is currently working on getting a file system API agreed upon which
> should satisfy this use case.

I think I did not explain my wish really well here; a file system API
is not really what I'm looking for. What I'm trying to do is to
"capture" some resources (product pictures or other multimedia data)
dynamically so that they are preferably served from within the
browser. The HTML5 application cache does not really help in this case
because the resources to capture have to be statically specified in a
manifest file. There is another W3C spec which addresses my concerns
really well: "Programmable HTTP Caching and Serving" (http://
www.w3.org/TR/DataCache/), but I'm not sure if any browser vendor
intends to implement it.

> You're welcome.  Hopefully the answers are better than you expected.

Thanks for taking the time! Your answers are great and clearly show
the direction of chrome's evolution with regard to the browsers
offlline features and local storage.

May I sneak in another really super technical question. In my tests
with the different sqlite based local db implementations I found that
in all of them it seems not possible to issue a VACUUM command to
compact the database. Probably (and hopefully) all implementations set
up the dbs with the auto_vacuum pragma, but the sqlite docs still say
that in the long run atuto_vacuum might "cause excess fragmentation of
the database file". So why not allow the VACUUM command?

[Jeremy Orlow]

Not sure, but I know this is one of the things Dumi was looking into at one point.  I think a lot of what you're suggesting here is not yet done simply due to resource constraints, not because we think they're bad ideas.

[John Dyer]

In the mean time, is there a way to break the 5MB web database limit
with a switch of some kind? I looked in
http://src.chromium.org/svn/trunk/src/chrome/common/chrome_switches.cc
but I can't find anything that would override the size limit.

On Feb 14, 5:08 pm, Jeremy Orlow <jor...@chromium.org> wrote:

[Jeremy Orlow]

Dumi, you're adding some UI for that, right?

--
You received this message because you are subscribed to the Google Groups "Chromium HTML5" group.
To post to this group, send email to chromiu...@chromium.org.
To unsubscribe from this group, send email to chromium-html...@chromium.org.
For more options, visit this group at http://groups.google.com/a/chromium.org/group/chromium-html5/?hl=en.

[Eric Uhrhane]

Resending from the right address...

On Tue, Feb 23, 2010 at 8:07 AM, Eric Uhrhane <er...@google.com> wrote:
> Sadly, no, there's no flag.  You'd have to build your own Chromium or
> WebKit to change the limits.

[PhistucK]

No, there is none, beside compiling on your own.

☆PhistucK

[Michael Nordman]

We started with something incredibly simple... a hard coded 5MB limit... of course we need to build in mechanisms to increase that limit with some user intervention. We don't know just what that will look like yet, but clearly it needs to be done and we'll get to it.

For right now... there is a no friendly way to increase the 5MB limit... but there is a way to do it.

Insert a row into the 'Quota' table for your 'origin' in this sqlite database file...

C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Default\databases\Databases.db

... the 'quota' value is in bytes. You can get the format of the 'origin' string value by looking at the 'origin' column of the 'Databases' table.

This means of hacking things will probably stop working after we figure out how to bump up quotas for real, but by that time there should be a much batter means of accomplishing this.