Disabling object persistence ?

[Marc'h]

Hello,

I'm battling with Django in order to try avoiding object persistence when initialising foreign key model attributes. The app I'm using Django for implements a facade for an already existing database. It's main purpose is to be used as a (read-only) ReST server returning objects built by aggregating attributes from the already existing database. The Django app itself has model classes (and associations) like any other Django app. Renderers and Serializers based on those supplied by the rest_framework modules have been written and tested. We started by configuring the Django app to use a SQLite file, as we're don't really care about object persistence. However, when building model instances (by extracting/aggregating data from the original non-Django managed database), we hit a severe perfomance bottleneck. Each time we initialize a model attribute which is a collection of related instances, all these are apparently persisted to the Django database.

For example, with a classical Order/Items association, we'd have :

01: orderattributes={...extract attributes from external database using SQL...}
02: order=Order(**orderattributes)
03: orderitems=[]
04: for item in ...items belonging to order...
05:    itemattributes={...extract attributes from external database using SQL...}
06:    item=Item(**itemattributes)
07:    orderitems.append(item)
08: order.items_set=orderitems

When reaching line 8, it seems that every Item instance of the orderitems collection is persisted in the Django database, which considerably slows down the application.

I tried to use an in-memory SQLite database, but couldn't make it work, as the syntax for declaring a shared in-memory database (file::memory:...) is not handled correctly, yielding a file-based database instead.

Thus, I'd like to know if there is any means to by-pass the automatic persistence mechanism when using collection attributes which are foreign-key based.

Thanks for any pointers !

Marc'h

[Daniel Roseman]

You have some fairly severe misunderstandings here. Django does not have any "automatic persistence mechanism". The *only* time a model instance is persisted to the database is when you call `save()` on it. I don't know what `order.items_set` is, but even assuming that it's a reverse foreign key manager, Django has no functionality that will automatically save those items - especially as the parent Order has not been saved, so therefore has no primary key for those FKs to link to. 

Part of the trouble here is that the code you've posted here (apart from being on a grey background, which makes it very hard to read) isn't your actual real code. You should post the actual code, and indicate where you think this "automatic persistence" is happening, then we can help you work out what really is going on.

--

DR.

[Javier Guerra Giraldez]

On Fri, Oct 4, 2013 at 1:45 AM, Marc'h <mhoe...@gmail.com> wrote:
> Thus, I'd like to know if there is any means to by-pass the automatic
> persistence mechanism when using collection attributes which are foreign-key
> based.

the ORM is a mapping to an SQL database, by definition that means
persistent storage. while you can create a model object and not
save() it, relationships are based on the record's primary key, which
is assigned (by the RDBMS) only when save()d.

so, no; there's no "direct" way to avoid persistence and use
relationships. in fact, any kind of query works only via the
database.

memory-only SQL tables would do the trick, but most implementations
use them only for connection-private tables. after a cursory re-read
of SQLite docs, I don't see any way to create a shared memory table,
the "file::memory:?cache=shared" URI only means that it is shared
between different connections on the same process

a not-so-nice option could be to specify a RAM disk (or tmpfs)
location for your database. it would sitll be file-based, and
persistant over different invocations, but faster and less durable
than a real disk file.

if not, then i think you should avoid the ORM and use a shared memory
storage; either memcached, Redis, uWSGI cache, or something similar.
it's not relational, but you should be able to create the needed
queries without too much trouble.

--
Javier