SQLite object error using hgvs 0.5

[Keith Callenberg]

Hi everyone,

I am receiving the following error upon calling variantmapper for most variants:

ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id <thread1> and this is thread id <thread2>.

I have installed hgvs 0.5.0a6 and am using a local UTA docker instance and have seqrepo installed per the installation notes: https://bitbucket.org/biocommons/hgvs/src/13be956259489dfe52ae94071591023d7cc6133d/notes/hgvs-0.5dev-install.rst.

Has anyone else seen this issue? Presumably it is something to do with the way I am using seqrepo since it uses SQLite and UTA does not. And the issue goes away when I reload hgvs after doing this:

del os.environ['HGVS_SEQREPO_DIR']

There is nothing obvious I am doing that is multi-threaded. I am running a bunch of hgvs calls at the same time, but all sequentially.

Thanks,

Keith

[Reece Hart]

Hi Keith-

I haven't seen that error before. I agree that this must be related to seqrepo.

Are you using threading at all? Neither hgvs nor seqrepo is explicitly threaded, so I'm a bit perplexed by the reported error.

Are you calling connect() more than once? Doing so is inefficient, but it should be okay. However, I really can't understand how'd see the behavior you're getting if connect is called only once because that's when SeqFetcher is constructed, which is the only time HGVS_SEQREPO_DIR is accessed.

-Reece 

[Peter Causey-Freeman]

Hi Reece and Keith.

I have had this error and have solved it. It is not caused specifically by SeqRepo. Are you using a package that uses multi-threading to call hgvs package and SeqRepo, e.g. a web micro-framework? 

If so, you will need to do a couple of things to get SeqRepo to work. 

1. Check that your install of SQLite3 (the actual SQLite3 not the python library) supports multi-threading. Most up-to-date versions do, however, you may need to re-install ensuring that your specific build has not been innstalled in a non-thread-safe way (This is an option and is, as far as I understand it, selected to speed SQLite3 up because the standard thread-safe install with no flags set is supposed to be slower). 

2. You will then need to ensure your Python install is directed to load the thread-safe version of SQLite3. I personally chose to re-install Python from scratch.

3. This is where it gets tricky. I do not use hgvs nor SeqRepo as a through Docker. I have them installed locally and have complete access to the install. I guess that this will be a decision for Reece to take. 

I adjusted the SeqRepo connection files as follows:

Python../site-packages/biocommons/seqrepo/seqaliasdb/seqaliasdb.py and change line 35 to         

self._db = sqlite3.connect(self._db_path, check_same_thread=False)

also line 69 of /site-packages/biocommons/seqrepo/fastadir.py needs to be set to

self._db = sqlite3.connect(self._db_path, check_same_thread=False).

The problem is that Reece will not be able to make sure that everyones build of SQLite has been installed in thread-safe mode, so there may not be a safe way for him to make this check_same_thread=False option available in the docker instance. If you do some looking around, you will notice that check_same_thread=False is under-documented because SQLite3 was historically not thread safe. 

Some folk will still criticise this fix, but I have been running this for about 2 months on variantvalidator.org with no problems what so ever. It has been rigorously tested using our variantvalidator.org/batchvalidator with several hundred thousand variants and it is proving to be a stable and fast option.

Hope this helps.

Pete

[Peter Causey-Freeman]

Actually Keith,

I read your post again. You should be able to make the updates I suggested because I think you have SeqRepo installed locally???. Feel free to contact me if you get stuck and I'll try to help. Reece should not have to touch the docker configs because, correct me if I'm wrong, they should use the resources of the docker virtual machine???

Hope this helps

Cheers guys

Pete

On Wednesday, January 11, 2017 at 10:06:39 PM UTC, Keith Callenberg wrote:

[Reece Hart]

On Fri, Jan 13, 2017 at 2:51 AM, Peter Causey-Freeman <pj...@le.ac.uk> wrote:

Reece should not have to touch the docker configs because, correct me if I'm wrong, they should use the resources of the docker virtual machine???

Yep, that's right.

Thanks for jumping in, Pete.

Any luck figuring this out, Keith?

-Reece

[Peter Causey-Freeman]

No problem Reece,

Hope the information helps.

Pete

[Reece Hart]

This issue was brought up again in https://github.com/biocommons/biocommons.seqrepo/issues/11. 

Keith or Liang- Would you please attach to the issue a simple example that triggers the problem? Thanks!

[Jake Peacock]

Hi Reece and Peter,

We have been running into this issue when utilizing celery to asynchronously access sqlite through seqrepo. We have tested in a thread-safe docker container and it works consistently well which is a huge improvement.

I'd love the ability to toggle this option check_thread_safe=False when connecting to sqlite through seqrepo settings.