Opening SQLite3 DB from StringIO
lis...@vitorbaptista.com
Is it possible? I have a somewhat large DB (~200 mb) and it would be
great if I could load it compressed (~3 mb) then, in memory,
decompress and load it, making my app less I/O-bound.
Thanks in advance,
Vítor Baptista.
Roger Binns
Hash: SHA1
On 07/10/2010 10:49 PM, lis...@vitorbaptista.com wrote:
> Is it possible? I have a somewhat large DB (~200 mb) and it would be
> great if I could load it compressed (~3 mb) then, in memory,
> decompress and load it, making my app less I/O-bound.
The easiest way of doing that would be to use a ramdisk. Under Linux you
can use tmpfs and there are many free ramdisks for Windows.
If you also want to conserve memory (ie not use 200mb of RAM with the
uncompressed copy) then you can license the SQLite CEROD extension:
http://www.hwaci.com/sw/sqlite/cerod.html
Alternately you can provide your own VFS implementation (you'll need APSW -
pysqlite doesn't have this functionality).
http://apsw.googlecode.com/svn/publish/vfs.html
The difficult part is mapping a request for an uncompressed offset/length
into where in the compressed data that is. The easiest way to implement
this would be to divide the data into blocks and compress each separately
with some sort of header giving the mapping offset and length for each
block. If you also want to allow writing then it gets even more tricky and
compressed block sizes will change as they are written to.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw5XukACgkQmOOfHg372QQmrACfcuqj7DBRrAip6tRmMsOMG85Z
8gwAoLb/oAMUxKhSFfOeRq/7ftjNLDHU
=ffUk
-----END PGP SIGNATURE-----