Static SQLite files

[Jim Cipar]

I'm trying to write an app that has to handle somewhat complex queries
to a never (or very rarely) changing data set. Right now I have a
static .csv file that my app reads, parses, and returns results from.
This solution isn't really ideal though, I would much rather have a
read-only SQLite file that the app can read, so that I can make use of
all of the good things that come with using a database (indices,
complex queries, performance ...).

It seems like App Engine refuses to load the sqlite3 python module,
even if I don't try to open a file for writing. Is there some
workaround to get read-only SQLite access in App Engine?

I can't imagine I'm the first person who's wanted to do this, but
everything I've found has been about people using SQLite as a local
store for the App Engine development kit, or wanting to use SQLite as
a read-write datastore.

[Felipe Teixeira]

You need use your SQLite in DataStore , use the api objectify its very simple.

bye

2011/7/18 Jim Cipar <jci...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Felipe Teixeira

Desenvolvedor Java

IPNET Soluções
Tel.: 55 21 3553 2717 / 21-7805-6867
Rua Visconde de Inhaúma, 134, Sala 615

Centro - RJ - CEP: 20091-007

http://www.ipnetsolucoes.com.br 

[Jim Cipar]

That doesn't solve either of my problems.

I'm not trying to use SQLite instead of DataStore, I'm trying to use
it instead of static CSV files. To paraphrase the SQLite
documentation: "don't think of SQLite as a replacement for MySQL,
think of it as a replacement for fopen()". I want a lightweight
structured file format for *static* data. Serving static data out of
DataStore is a waste of resources.

Secondly, DataStore doesn't solve the complex query problem. It can't
do "JOIN", "GROUP BY", or any kind of nested queries.

Lastly, a memory-only SQLite instance is a very useful abstraction for
doing data transformations even when you're not worried about
persistence at all. Even if we could simply load the SQLite module
for memory-only databases that would be a step in the right direction,
though storing my static data in SQLite format would be ideal.



On Jul 18, 10:25 am, Felipe Teixeira

> *Felipe Teixeira*
> *Desenvolvedor Java
> *

> IPNET Soluções
> Tel.: 55 21 3553 2717 / 21-7805-6867
> Rua Visconde de Inhaúma, 134, Sala 615
> Centro - RJ - CEP: 20091-007

> *http://www.ipnetsolucoes.com.br***
>
> **
> *
> *

[Barry Hunter]

http://code.google.com/appengine/docs/python/runtime.html#Pure_Python

Can't load C extensions.

http://www.google.com/search?q=sqlite+pure+python
Might lead to something useful...?

[Gary Frederick]

would using fusion tables in app engine do what you want?

  http://code.google.com/apis/fusiontables/docs/sample_code.html

I do NOT know, but it looks like you can use fusion tables in app engine and fusion tables provide SQLite queries.

or I could be completely wrong - in which case - never mind...

Gary 

[Tim Hoffman]

Hi Jim

You won't be able to load the sqlite3 module in python on appengine as it links to a 'binary module' (ie sqlite is not implemented in pure python.)

You will need to rethink your strategy here if you want to use appengine

Regards

Tim

[刘浩]

sorry you shold say something easy my English is not good

2011/7/19 Tim Hoffman <zute...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dNH4diB9SLsJ.

[Jonny Bergström]

I know this is ages later, but I am constantly having the same "need"; "if only I could at least read sqlite files in memory". I've been fiddling with trying to get a pure python sqlite implementation running and what not but never succeeded at anything. Being able to at least read sqlite files would make GAE much more useful to me.

[Patrice (Cloud Platform Support)]

Hi Johnny,

While I can't promise anything, you would have more traction for such a request by posting it on our issue tracker, here.

Post this with as much information as you can, and we'll follow up there to see what can be done.

Cheers, and have a good weekend!

[Tomasz Jamróz]

Hi,
I am facing similar problem and found out that the most probable reason is sandbox.py not having '_sqlite3' in _WHITE_LIST_C_MODULES.
After adding _sqlite3 to _WHITE_LIST_C_MODULES in my local sandbox.py I managed to both import sqlite3 and execute queries to the database with script running on GAE on my local machine. After deployment, however, the server threw 500 error. Is it possible that you fix this problem? I understand that your policy is to encourage users to use Datastore, but in some cases using sqlite3 is really needed for reasons of compatibility with other apps already existing.
Thanks


[Alex Martelli]

When you want to run in App Engine but also need some modules or extension that the App Engine sandbox does not support, consider using managed VMs (MVMs).

For example, here's a Python App Engine version of the usual guestbook example app using exactly sqlite for its storage layer.

It's important to note that, if your App Engine app is properly structured into modules, as Google recommends (if you think you're not using modules, you are anyway: what you think as "your app" is actually "your app's default module" and you're just currently choosing not to add others), you can mix and match traditional ("sandboxed", thus unable to use certain modules and extensions such as sqlite) and MVM-hosted modules, for maximum flexibility.

The only key thing to keep in mind is that managed VMs are in beta: thus, perfectly suitable for development and experimenting, but not yet for production deployment at scale.

Requesting a new feature for traditional (sandboxed) App Engine, that's already present in MVMs, as Patrice suggested (such as, the ability to use sqlite, or some other module or extensions not currently supported in the sandbox), is clearly possible.

However, such a new feature request, if and when accepted, will also have to go through its own period of development, testing, and then beta release, before it's released as "generally available" (out of beta, thus fully supported and recommended for production deployments).

Therefore, even if you do choose to open such a feature request, I would recommend also, at the same time, trying out the deployment of an experimental version of your app (ideally, only the one new module requiring sqlite3, with the other modules remaining sandboxed) on MVMs -- so you'll be ready for production deployment when any positive news emerges, one way or another.

Yet another workaround to help tide you over is Cloud SQL -- it's MySQL, not sqlite3, but migration between SQL dialects is usually much easier than moving to NoSQL DBs such as the App Engine datastore; and, Cloud SQL is fully supported (not beta!). So, you could immediately deploy the "database access module" in production in a sandboxed variant using Cloud SQL, while also developing and experimenting with a MVM-deployed variant for the time when, one way or another, sqlite3 will be available in the way you prefer and fully supported for production use.

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/dff00677-9e80-4b0d-acd9-789c9be3c773%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tomasz Jamróz]

Thanks Alex! That was very helpful!
Best,
Tomasz