In-memory queryset

262 views
Skip to first unread message

pa...@dabapps.com

unread,
Mar 7, 2017, 9:57:57 AM3/7/17
to Django developers (Contributions to Django itself)
Hi,

It would be really convenient for me if there was an implementation of the QuerySet API which instead of using a database as its data source, used in-memory model instance that had not been persisted to the database at all.

I looked around and found nothing like this.

Is this because nothing like this exists? Is it because it's a terrible idea for a reason that's not obvious to me? Would it not be awesome for the API for interacting with collections of in-memory instances and with a database-backed collection to be the same?

Many thanks,

Paul

ludovic coues

unread,
Mar 7, 2017, 10:10:21 AM3/7/17
to django-d...@googlegroups.com
This look like a question for django user.

The QuerySet API is only an abstraction layer to SQL code. That's why
it require a database.
If you really don't want to store your object on a filesystem, sqlite
support database living only in memory. That's mainly for testing
purpose but that could fit your use-case. And django support multiple
database.
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d4f7c142-5d3f-4158-98bb-1e2171f0e63e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Cordialement, Coues Ludovic
+336 148 743 42

Jacob Kaplan-Moss

unread,
Mar 7, 2017, 10:17:55 AM3/7/17
to django-developers
Hey Paul -

There are a couple of implementations I'm aware of:


Neither are a complete implementation of the QuerySet API, and I think this is because the QuerySet API surface is _huge_. Implementing the entire API would be a monumental task (especially if you start wanting to implement some of the more complex operations, like annotate/aggregate). At a certain point, it because easier to just use a real database -- most people use SQLite at that point.

It's certainly _possible_ to implement a in-memory datastore, and it might be pretty useful. It's just pretty dang hard to do more than the first 20% or so.

Jacob

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Tim Chase

unread,
Mar 7, 2017, 11:16:32 AM3/7/17
to django-d...@googlegroups.com
On 2017-03-07 07:17, Jacob Kaplan-Moss wrote:
> It's certainly _possible_ to implement a in-memory datastore, and
> it might be pretty useful. It's just pretty dang hard to do more
> than the first 20% or so.

Would there also be issues with WSGI spinning up/down various Django
processes? Any in-memory (in-process) data would be lost when the
process goes down. I could see it working for an individual dev
server or if you could guarantee that a single WSGI process remains
running indefinitely. But for those, I'd just go with sqlite's
":memory:" option.

-tkc



Neal Todd

unread,
Mar 8, 2017, 7:19:18 AM3/8/17
to Django developers (Contributions to Django itself)
Hi Paul,

Depending on your use cases django-modelcluster might be of some use:

https://github.com/wagtail/django-modelcluster

It's primarly for dealing with a bunch of models that have foreign key relationships, but before they're written to a database (i.e before they have primary keys that allow the foreign keys to work). At some point they are typically committed to a database but they also could be ephemeral or serialised and handed off to something else.

It only supports a subset of the QuerySet API (pragmatically that which allows it to serve the purpose it was created for).

Cheers, Neal  
Reply all
Reply to author
Forward
0 new messages