Support for pysqlcipher3

[Fayaz Yusuf Khan]

Hi,

pysqlcipher3 is a Python3 port for pysqlcipher (for SQLite). Does SQLA currently have support for this? (I wasn't able to get it running for my project.)
If not, any chances it will, in the future?

Thanks.

[Mike Bayer]

pysqlcipher is right here:

http://docs.sqlalchemy.org/en/rel_1_0/dialects/sqlite.html#module-sqlalchemy.dialects.sqlite.pysqlcipher

SQLAlchemy supports Python 2K and Python 3K in place though the above
dialect has probably not been tested on Py3K.


Which means. "I wasn't able to get it running" == ask for help w/ your
specific issue and/or report a bug.

[Fayaz Yusuf Khan]

On Sunday 31 May 2015 11:06:32 AM Mike Bayer wrote:
> pysqlcipher is right here:
>
> http://docs.sqlalchemy.org/en/rel_1_0/dialects/sqlite.html#module-sqlalchemy
> .dialects.sqlite.pysqlcipher
>
> SQLAlchemy supports Python 2K and Python 3K in place though the above
> dialect has probably not been tested on Py3K.

pysqlcipher does not support Py3K.
https://github.com/leapcode/pysqlcipher/issues/3
It's a known issue.

[Mike Bayer]

you can send in "pysqlcipher3" to create_engine() using the "dbapi"
argument:

import pysqlcipher3
e = create_engine("sqlite+pysqlcipher:///file.db", dbapi=pysqlcipher3)

feel free to submit a PR that returns this DBAPI from the "dbapi" method
under py3k:


https://bitbucket.org/zzzeek/sqlalchemy/src/0766c80b9c02fdbad3203835ab850ad690f4c03b/lib/sqlalchemy/dialects/sqlite/pysqlcipher.py?at=master#cl-82


>

[Fayaz Yusuf Khan]

Alright, thanks!
I would try to submit the PR. But it does look like a transitional package to me. How do we generally handle such upstream changes anyway?

[Fayaz Yusuf Khan]

On Sunday 31 May 2015 11:36:12 PM Mike Bayer wrote:

> you can send in "pysqlcipher3" to create_engine() using the "dbapi"
> argument:
>
> import pysqlcipher3
> e = create_engine("sqlite+pysqlcipher:///file.db", dbapi=pysqlcipher3)
>
> feel free to submit a PR that returns this DBAPI from the "dbapi" method
> under py3k:

So, I've been trying to make SQLiteDialect_pysqlcipher.dbapi return
pysqlcipher3.dbapi2.

But before I do that, I had to ensure all the tests were passing in the first
place. But when I run the tests like this:

py.test --dburi=sqlite+pysqlcipher://:test@/test.db -x

I'm seeing:

...
test/aaa_profiling/test_memusage.py::MemUsageTest_sqlite_pysqlcipher::test_fixture
PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_sqlite_pysqlcipher::test_join_cache
FAILED

============================================================== FAILURES
==============================================================
__________________________________________
MemUsageTest_sqlite_pysqlcipher.test_join_cache
___________________________________________
Traceback (most recent call last):
File "<string>", line 2, in test_join_cache
File
"/home/fayaz/Programming/sqlalchemy/test/../lib/sqlalchemy/testing/exclusions.py",
line 94, in decorate
return self._do(config._current, fn, *args, **kw)
File
"/home/fayaz/Programming/sqlalchemy/test/../lib/sqlalchemy/testing/exclusions.py",
line 123, in _do
self._expect_failure(config, ex, name=fn.__name__)
File
"/home/fayaz/Programming/sqlalchemy/test/../lib/sqlalchemy/testing/exclusions.py",
line 135, in _expect_failure
util.raise_from_cause(ex)
File
"/home/fayaz/Programming/sqlalchemy/test/../lib/sqlalchemy/util/compat.py",
line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File
"/home/fayaz/Programming/sqlalchemy/test/../lib/sqlalchemy/testing/exclusions.py",
line 121, in _do
return_value = fn(*args, **kw)
File
"/home/fayaz/Programming/sqlalchemy/test/aaa_profiling/test_memusage.py", line
732, in test_join_cache
go()
File
"/home/fayaz/Programming/sqlalchemy/test/aaa_profiling/test_memusage.py", line
96, in profile
assert False, repr(samples)
AssertionError: [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654,
227655, 227656, 227657, 227658, 227659, 227660, 227661, 227662, 227663,
227664, 227665, 227666, 227667, 227668, 227669, 227670, 227671, 227672,
227673, 227674, 227675, 227676, 227677, 227678, 227679]
assert False
-------------------------------------------------------- Captured stdout call
--------------------------------------------------------
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654,
227655, 227656, 227657, 227658, 227659])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654,
227655, 227656, 227657, 227658, 227659, 227660, 227661, 227662, 227663,
227664])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654,
227655, 227656, 227657, 227658, 227659, 227660, 227661, 227662, 227663,
227664, 227665, 227666, 227667, 227668, 227669])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654,
227655, 227656, 227657, 227658, 227659, 227660, 227661, 227662, 227663,
227664, 227665, 227666, 227667, 227668, 227669, 227670, 227671, 227672,
227673, 227674])
('sample gc sizes:', [227630, 227631, 227632, 227633, 227634, 227635, 227636,
227637, 227638, 227639, 227640, 227641, 227642, 227643, 227644, 227645,
227646, 227647, 227648, 227649, 227650, 227651, 227652, 227653, 227654,
227655, 227656, 227657, 227658, 227659, 227660, 227661, 227662, 227663,
227664, 227665, 227666, 227667, 227668, 227669, 227670, 227671, 227672,
227673, 227674, 227675, 227676, 227677, 227678, 227679])
====================================================== short test summary info
=======================================================
FAIL test/aaa_profiling/test_memusage.py::MemUsageTest_sqlite_pysqlcipher::
()::test_join_cache

[Mike Bayer]

On 6/13/15 5:08 AM, Fayaz Yusuf Khan wrote:
> On Sunday 31 May 2015 11:36:12 PM Mike Bayer wrote:
>
>> you can send in "pysqlcipher3" to create_engine() using the "dbapi"
>> argument:
>>
>> import pysqlcipher3
>> e = create_engine("sqlite+pysqlcipher:///file.db", dbapi=pysqlcipher3)
>>
>> feel free to submit a PR that returns this DBAPI from the "dbapi" method
>> under py3k:
> So, I've been trying to make SQLiteDialect_pysqlcipher.dbapi return
> pysqlcipher3.dbapi2.
>
> But before I do that, I had to ensure all the tests were passing in the first
> place. But when I run the tests like this:
>
> py.test --dburi=sqlite+pysqlcipher://:test@/test.db -x
>
> I'm seeing:
>
> ...
> test/aaa_profiling/test_memusage.py::MemUsageTest_sqlite_pysqlcipher::test_fixture
> PASSED
> test/aaa_profiling/test_memusage.py::MemUsageTest_sqlite_pysqlcipher::test_join_cache
> FAILED

if that is the only test failing I wouldn't worry too hard. if you
look at the test, you'll see this comment:

# fails on newer versions of pysqlite due to unusual memory behvior
# in pysqlite itself. background at:
# http://thread.gmane.org/gmane.comp.python.db.pysqlite.user/2290

In fact there's some hardcoded logic to help with this test that is
hardcoded to pysqlite, try this patch:

--- a/test/aaa_profiling/test_memusage.py
+++ b/test/aaa_profiling/test_memusage.py
@@ -45,7 +45,8 @@ def profile_memory(maxtimes=50):
# tests under 50 iterations and ideally about ten, so
# just filter them out so that we get a "flatline" more
quickly.

- if testing.against("sqlite+pysqlite"):
+ if testing.against("sqlite+pysqlite") or \
+ testing.against("sqlite+pysqlcipher"):
return [o for o in gc.get_objects()
if not isinstance(o, weakref.ref)]
else:

[Fayaz Yusuf Khan]

On Saturday 13 Jun 2015 7:42:51 AM Mike Bayer wrote:
> if that is the only test failing I wouldn't worry too hard. if you
> look at the test, you'll see this comment:
>
> # fails on newer versions of pysqlite due to unusual memory behvior
> # in pysqlite itself. background at:
> # http://thread.gmane.org/gmane.comp.python.db.pysqlite.user/2290
>
> In fact there's some hardcoded logic to help with this test that is
> hardcoded to pysqlite, try this patch:
>
> --- a/test/aaa_profiling/test_memusage.py
> +++ b/test/aaa_profiling/test_memusage.py
> @@ -45,7 +45,8 @@ def profile_memory(maxtimes=50):
> # tests under 50 iterations and ideally about ten, so
> # just filter them out so that we get a "flatline" more
> quickly.
>
> - if testing.against("sqlite+pysqlite"):
> + if testing.against("sqlite+pysqlite") or \
> + testing.against("sqlite+pysqlcipher"):
> return [o for o in gc.get_objects()
> if not isinstance(o, weakref.ref)]
> else:

What about these?

====================================================== short test summary info
=======================================================

FAIL test/dialect/test_sqlite.py::InsertTest::()::test_empty_insert_pk4
FAIL test/dialect/test_suite.py::ComponentReflectionTest_sqlite_pysqlcipher::
()::test_get_table_names
FAIL test/dialect/test_suite.py::ComponentReflectionTest_sqlite_pysqlcipher::
()::test_get_table_names_fks
FAIL test/dialect/test_suite.py::ComponentReflectionTest_sqlite_pysqlcipher::
()::test_get_tables_and_views
FAIL test/engine/test_reflection.py::ReflectionTest_sqlite_pysqlcipher::
()::test_reflect_all
FAIL test/engine/test_reflection.py::ReflectionTest_sqlite_pysqlcipher::
()::test_reflect_all_with_views
FAIL test/engine/test_reflection.py::UnicodeReflectionTest_sqlite_pysqlcipher::
()::test_basic
FAIL test/engine/test_reflection.py::UnicodeReflectionTest_sqlite_pysqlcipher::
()::test_get_names
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 25
failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================== 8 failed, 3115 passed, 611 skipped, 17
error in 1919.33 seconds ===================================

[Mike Bayer]

On 6/14/15 7:29 AM, Fayaz Yusuf Khan wrote:
> On Saturday 13 Jun 2015 7:42:51 AM Mike Bayer wrote:
>> if that is the only test failing I wouldn't worry too hard. if you
>> look at the test, you'll see this comment:
>>
>> # fails on newer versions of pysqlite due to unusual memory behvior
>> # in pysqlite itself. background at:
>> # http://thread.gmane.org/gmane.comp.python.db.pysqlite.user/2290
>>
>> In fact there's some hardcoded logic to help with this test that is
>> hardcoded to pysqlite, try this patch:
>>
>> --- a/test/aaa_profiling/test_memusage.py
>> +++ b/test/aaa_profiling/test_memusage.py
>> @@ -45,7 +45,8 @@ def profile_memory(maxtimes=50):
>> # tests under 50 iterations and ideally about ten, so
>> # just filter them out so that we get a "flatline" more
>> quickly.
>>
>> - if testing.against("sqlite+pysqlite"):
>> + if testing.against("sqlite+pysqlite") or \
>> + testing.against("sqlite+pysqlcipher"):
>> return [o for o in gc.get_objects()
>> if not isinstance(o, weakref.ref)]
>> else:
> What about these?

if these all work with the default pysqlcipher then it's likely these
are bugs of some kind in whatever version you're working with, not sure.