sqlite3, memory db and multithreading
królewna
least 2 threads have to have access to in-memory sqlite database. It is
not possible to pass sqlite objects to those threads because an
exception is rised:
ProgrammingError: SQLite objects created in a thread can only be used in
that same thread.The object was created in thread id -1219066176 and
this is thread id -1224475792
Is there any EASY way to use this in-memory db in many threads? Creating
another connection is not a solution as it creates completely new db
instead of connecting to the existing one.
--
Best regards
princess
DreiJane
i cannot help you directly with sqlite2 in the Standardlib, since i am
used to work with Roger Binns's apsw. After a short experiment with
pysqlite leading to data loss - caused by one of the unclearer
exception messages of sqlite3 and me having a bad day - i at once
turned back to apsw. And so far i haven't done much with threads.
Principally sqlite connections (sqlite3 objects in the C-API) can be
used over multiple threads - and connections to :memory: make no
difference. There are additional parameters to open() giving fine-
tuned control. And apsw is promising a true reflection of sqlite's C-
API.
Regards, Joost
Aahz
You probably need to serialize access to the database through one thread.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer
John Nagle
Recognize that sqlite is for "lite" database work. If you're running
some massively concurrent database application, you need something heavier,
like MySQL or Postgres. "sqlite" has a simplistic locking strategy.
Locking is done by file-level locking, and you can have one UPDATE/INSERT
operations, or any numnber of SELECTs, at a time. Lock conflicts are
handled by wait and retry, which is slow.
The big databases are much smarter about figuring out which operations
can safely be done in parallel, do much more query optimization, and
handle high transaction volumes much better than sqlite.
You use sqlite for configuration files, your personal databases, and
other small stuff. You run your Web 2.0 site on MySQL or Postgres.
You run your Fortune 1000 company on Oracle.
John Nagle
Expo
You can put the SQLite database into a Singleton class and use a
semaphore to serialize the access to methods which writes to the
database.
królewna
> Principally sqlite connections (sqlite3 objects in the C-API) can be
> used over multiple threads - and connections to :memory: make no
> difference. There are additional parameters to open() giving fine-
> tuned control. And apsw is promising a true reflection of sqlite's C-
> API.
It's my fault not saying what am I using to connect to sqlite db. It's
sqlite3 module. So there is no open() function etc. As for now I'm not
interested in rewriting my program to use apsw so is there any possible
way of working this out with sqlite3?
--
best regards
princess
królewna
>
> You probably need to serialize access to the database through one thread.
sqlite3 objects are not pickable so it's not proper way.
--
best regards
princess
królewna
>
> You can put the SQLite database into a Singleton class and use a
> semaphore to serialize the access to methods which writes to the
> database.
I've tried this out but doesnt work. Still gives an error like:
ProgrammingError: SQLite objects created in a thread can only be used in
that same thread.The object was created in thread id -1216280896 and
this is thread id -1217107088
--
best regards
princess
Tim Golden
> W dniu 18.03.2010 23:06, Aahz pisze:
>>
>> You probably need to serialize access to the database through one thread.
>
> sqlite3 objects are not pickable so it's not proper way.
Is it possible you've misunderstood the meaning of the word "serialize"
here? What's being suggested isn't serialising (ie marshalling, pickling)
the data; rather, serialising the *access*, ie pushing all db requests into
a queue which is read by one thread which manages the only db connection.
TJG
królewna
> Is it possible you've misunderstood the meaning of the word "serialize"
> here?
> What's being suggested isn't serialising (ie marshalling, pickling)
> the data; rather, serialising the *access*, ie pushing all db requests into
> a queue which is read by one thread which manages the only db connection.
That would make structure of program much more complicated. I would have
to create queue for putting there queries and some other extra
variables/structure to receive output from db and some more for
controlling the execution flow of awaiting threads.
--
best regards
princess
królewna
is not a single word about this option. I've found it in pysqlite doc
site. So we have to add a new keyword argument to connection function
and we will be able to create cursors out of it in different thread. So use:
sqlite.connect(":memory:", check_same_thread = False)
works out perfectly for me. Of course from now on me need to take care
of safe multithreading access to the db. Anyway thx all for trying to help.
--
best regards
princess