Re: Djonet, the MonetDB backend for Django

[Michael Manfre]

I haven't tested your patch or backend and only have a few comments on implementation ideas for a custom database backend.

On Monday, July 9, 2012 11:25:47 AM UTC-4, Gijs Molenaar wrote:

These problems are:

 * MonetDB supports a Decimal type with a maximum length of 18 digits.

Override DatabaseOperations.value_to_db_decimal and clamp the max_digits to 18. I'm not sure if you'll need other modifications.
 

 * No support for hyphens in column names

Hyphens are not allowed even when the column name is quoted? If so, strip then you can strip them out in DatabaseOperations.quote_name.
 

 * No support for != operation (only <>).

You'll need to provide your own SQLCompiler and do the substitution there. You can look at django-mssql's implementation [1] as an example.

[1] https://bitbucket.org/Manfre/django-mssql/src/eff1ed275fdd/sqlserver_ado/compiler.py

Regards,
Michael Manfre

[Gijs Molenaar]

hi Michael,

Thanks for your response! One more question, see below.

Op dinsdag 10 juli 2012 17:51:16 UTC+2 schreef Michael Manfre het volgende:

I haven't tested your patch or backend and only have a few comments on implementation ideas for a custom database backend.

[...]

 

 * No support for != operation (only <>).

You'll need to provide your own SQLCompiler and do the substitution there. You can look at django-mssql's implementation [1] as an example.

But the test script has != hardcoded in it, is it the idea that you scan every query for such character combination? Isn't this bad practice since it will have a performance impact since it is performed on all queries?

[Michael Manfre]

On Thursday, July 19, 2012 10:15:50 AM UTC-4, Gijs Molenaar wrote:

hi Michael,

Thanks for your response! One more question, see below.

But the test script has != hardcoded in it, is it the idea that you scan every query for such character combination? Isn't this bad practice since it will have a performance impact since it is performed on all queries?

Correct. You need to check and fix all queries. The overhead of a single, compiled regex replace per query should be negligible. The django-mssql backend does a lot more lifting than that for any query with aggregates or slicing.

Regards,
Michael Manfre

[Gijs Molenaar]

Hi Michael,

Op dinsdag 10 juli 2012 17:51:16 UTC+2 schreef Michael Manfre het volgende:

On Monday, July 9, 2012 11:25:47 AM UTC-4, Gijs Molenaar wrote:

These problems are:

 * MonetDB supports a Decimal type with a maximum length of 18 digits.

Override DatabaseOperations.value_to_db_decimal and clamp the max_digits to 18. I'm not sure if you'll need other modifications.
 

This doesn't seem to solve the problem. This only limits the convertion of Decimal types to 18 digits, but not the creation of the table itself. I Think the correct place to define such a things would be do something with DatabaseCreation.data_types. Any suggestions?

Thanks!

 - Gijs

[Michael Manfre]

DatabaseCreation.data_types allows you to set a format string that should be used for the various types. I'm not sure how you could use that to change either max_digits or decimal_places to clamp them for MonetDB. You could monkey patch DecimalField or provide your own field that enforces the DB's limit, but I think it would be easier from a maintenance standpoint to document the limitation. People have full control over the models that they use in their projects, including from 3rd party apps. Silently changing the database schema is probably not a good policy to take.

--
Michael Manfre

[Gijs Molenaar]

Op maandag 30 juli 2012 16:05:14 UTC+2 schreef Michael Manfre het volgende:

Ok, well then probably indeed it is best to do it this way. Probably it is better to confront the user with an exception than to implicitly change it's datatype anyway. Thanks for the help.

If you don't mind I have an other question, I hope you have the time to share your thoughts on this one also. I'm digging trough all the test cases and fixing them, but i'm a bit stuck on this one. test_forward_refs (modeltests.serializers.tests.JsonSerializerTransactionTestCase) fails in my case, generating a foreign key constrain violation. I've set supports_forward_references to False in features, but doesn't seem to have any effect. Do you have any interesting suggestion?

Thanks!

 - Gijs

[Michael Manfre]

On Tue, Jul 31, 2012 at 6:50 AM, Gijs Molenaar <gijsmo...@gmail.com> wrote:

If you don't mind I have an other question, I hope you have the time to share your thoughts on this one also. I'm digging trough all the test cases and fixing them, but i'm a bit stuck on this one. test_forward_refs (modeltests.serializers.tests.JsonSerializerTransactionTestCase) fails in my case, generating a foreign key constrain violation. I've set supports_forward_references to False in features, but doesn't seem to have any effect. Do you have any interesting suggestion?

Thanks!

 - Gijs

I vaguely remember encountering a problem with forward references when I was adding support to django-mssql for Django 1.4. Can you post the traceback? It might jog my memory.

--
Michael Manfre

[Gijs Molenaar]

Hi Michael,

I see now that most of the Django tests that are still failing seem to be forward referencing problems. Django inserts a record to a table with a reference to something that doesn't exists (yet). I find it a bit hard to debug, since I've not managed to disable transactions while running the tests (setting autocommit in the settings file as no effect) and thus can not query the database. 

Here is one stacktrace:

"""

/usr/bin/python2.7 /home/gijs/Work/django/tests/runtests.py --settings testdjonet.settings --failfast serializers

Creating test database for alias 'default'...

created database in maintenance mode: test_djonet

taken database out of maintenance mode: test_djonet

starting database 'test_djonet'... done

........................Destroying test database for alias 'default'...

E

======================================================================

ERROR: test_forward_refs (modeltests.serializers.tests.JsonSerializerTransactionTestCase)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/gijs/Work/django/tests/modeltests/serializers/tests.py", line 275, in test_forward_refs

    obj.save()

  File "/home/gijs/Work/django/django/core/serializers/base.py", line 165, in save

    models.Model.save_base(self.object, using=using, raw=True)

  File "/home/gijs/Work/django/django/db/models/base.py", line 551, in save_base

    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "/home/gijs/Work/django/django/db/models/manager.py", line 203, in _insert

    return insert_query(self.model, objs, fields, **kwargs)

  File "/home/gijs/Work/django/django/db/models/query.py", line 1576, in insert_query

    return query.get_compiler(using=using).execute_sql(return_id)

  File "/home/gijs/Work/django/django/db/models/sql/compiler.py", line 910, in execute_sql

    cursor.execute(sql, params)

  File "/home/gijs/Work/monetdb/MonetDB/clients/python/monetdb/sql/cursors.py", line 192, in execute

    block = self.connection.execute(query)

  File "/home/gijs/Work/monetdb/MonetDB/clients/python/monetdb/sql/connections.py", line 123, in execute

    return self.command('s' + query + ';')

  File "/home/gijs/Work/monetdb/MonetDB/clients/python/monetdb/sql/connections.py", line 128, in command

    return self.mapi.cmd(command)

  File "/home/gijs/Work/monetdb/MonetDB/clients/python/monetdb/mapi.py", line 171, in cmd

    raise OperationalError(response[1:])

OperationalError: M0M29!INSERT INTO: FOREIGN KEY constraint 'serializers_article.serializers_article_author_id_fkey' violated

"""

Op maandag 9 juli 2012 17:25:47 UTC+2 schreef Gijs Molenaar het volgende:

Hi Django dev,

I'm working on a MonetDB backend for Django called djonet:

https://github.com/gijzelaerr/djonet

Feel free to test it and let me know if it works for you. I think most stuff  is sort of working, but when I run the Django test suite with a djonet settings some errors pop up that I don't really know yet how to solve.

These problems are:

 * MonetDB supports a Decimal type with a maximum length of 18 digits.

 * No support for hyphens in column names

 * No support for != operation (only <>).

For now I just modified the Django test suite (see attachment), but that doesn't seem to be very constructive. Any suggestion on how to solve these problems?

thanks,

 - Gijs