H2 as a big disk-based cache

[Driss]

Hi all,

I'll try to give as much background as possible so
that people can point me in the correct direction.

I currently have a Java application running on a lot
of users machine (hundreds, soon thousands).
One of the thing I do really need is a big cache/lookup
(which takes a *very* long time to compute and which
is done only once, upon installing the application) table
which has approx 60 million consecutive entries being
19 bits each (about 135 MB).

It is really needed and as of now I'm keeping it
entirely in-memory using my own "bit packing".
The problem is that 135 MB in ram is too much of a
burden on some of my users machines. And it's not
really needed in memory: it's not queried that often.

So I'd like to put these entries on disk using H2 if
it's a good idea (instead of doing it all myself), knowing
that my main concern is the lowest possible memory
usage by H2.

However I'm a bit lost: I checked the documentation
but didn't really "get it".

If I wanted to use H2 to store these 60 million
entries and if my main goal was the smallest possible
H2 memory usage, how should I go about configuring
/ launching embedded H2 and what kind of schema
should I be using to store these entries? (knowing that
bit-manipulation hackery ain't an issue, that's how I'm
doing it now).

[Peter Zhang]

check this: http://www.h2database.com/html/faq.html#database_files
If you wanna limit your memory, add -X for your JVM

--

You received this message because you are subscribed to the Google Groups "H2 Database" group.
To post to this group, send email to h2-da...@googlegroups.com.
To unsubscribe from this group, send email to h2-database...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/h2-database?hl=en.

--
    Best Regard

zozoh @ 2009
-----------------------------------------

[Sam Van Oort]

Hi Driss,

H2 should work well for you (the defaults are very conservative with
memory). To get started with H2, I suggest you consult the excellent
tutorial here: http://www.h2database.com/html/tutorial.html

Since this is an unusual request, I think we can help you better if
you answer a couple questions:

What sort of data are you storing (I'm guessing bytes/integers), and
is there any duplication within the table?

Is your lookup table changed at all after you first compute it? If
not, you can use a read-only table, maybe even a zipped one.

Are you concerned about limiting disk space used to store the table,
and if so, how small would the DB need to be?

When you say "not queried that much" how fast does the query need to
run? This determines what settings will be used to store it and how
indices will be used.

Optional: what is your favorite local food? (H2 has users around the
world, and I'm curious what treats you guys enjoy.)

Cheers,
Sam Van Oort
(Junior Committer to H2 project)

> > h2-database...@googlegroups.com<h2-database%2Bunsu...@googlegroups.com>

[Thomas Mueller]

Hi,

I would use a regular (persistent) table. Maybe it's faster if you
just use 32 bit per entry, maybe you could group the entries into
groups of 800 entries. If you do that, then each group is 15200 bits
large, which would be good for the new 'page store' algorithm (it uses
a page size of 2 KB by default; each page needs about 100 bytes
overhead; with each group being 1900 bytes you are close to the
optimum). I'm not sure if you can easily group the entries however.

Regards,
Thomas

> --
>
> You received this message because you are subscribed to the Google Groups "H2 Database" group.
> To post to this group, send email to h2-da...@googlegroups.com.

> To unsubscribe from this group, send email to h2-database...@googlegroups.com.