Re: [edx-code] Accessing edX datbases from external Django project

[Yarko Tymciurak]

On Mon, Jan 6, 2014 at 6:59 AM, Javier Santofimia Ruiz <javi...@gmail.com> wrote:

Hi! I'm working on an external project using Django that take data from edX platform and analyze it to show some graphics. I'm facing 2 problems here:

  - How to access to a external database from Django: I think I can do that following these instructions: https://docs.djangoproject.com/en/dev/topics/db/multi-db/ . But I only want to read data, and don't want to clone the database, ¿this will work?

connecting to a database server doesn't require a clone - only a connection.

sqlite doesn't use a server, only a file. 

  - Which are the credentials to access MySQL/SQLite and MongoDB databases. If the user data is an external MySQL this will be easy (just take the host and name from that external database), but in my case, the edX platform I've mounted in my system is for development purposes, so the database for user data is a SQLite database (am I wrong?). I've found that edx project define SQLite database like this:  "'NAME': ENV_ROOT / "db" / "mitx.db"" , but how can I access it from outside the project? About the MongoDB access, I've no idea how can I access to it, only thing I've seen is that the database is not defined in DATABASE field in edx project but in MODULESTORE field.

BTW, the db name is "edx.db" in the current tree... 

If you are working from your "host" / development computer, then the solution would seem straightforward:

Since sqlite uses a file (not a server) for db access, and since the VM mounts space from your host machine to run the edx code,  simple define a [lms,cms]/envs/  file which places the 'NAME' of the edx.db file someplace into the mounted directory (and therefore into your host computer).

For example,   in lms/envs/devstack_db_anal.py:

---------------------------------------

from .devstack.py import *

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.sqlite3',

        'NAME': REPO_ROOT / "db" / "edx.db",

    }

}

------------------

Note that 'NAME'  changed from ENV_ROOT to REPO_ROOT.

These are set in envs/common.py.

You might find a more suitable location for your tests.

This should set the path for the sqlite database to edx-platform/db/edx.db - you will need to make a directory "edx-platform/db" on your development machin in your repository to accomodate this.   This should now be where the VM edx instance of lms references the database.  You will also want to do a similar change for cms.   You can then refer to that path from your "other" django project on your development machine also.

Hope this suggestion is helpful.

Regards,

Yarko

 

Hope you can help me, thank you all, and sorry for my English!!!

PS: I know about Insights module, but I don't want to use it until the project is more stable.

[Javier Santofimia Ruiz]

Hi! Thanks for your answer!
 
-About the db name, yes I had an older version of the platform installed (I installed it 2 months ago). I've just made a fresh install with the method described in https://github.com/edx/configuration/wiki/edX-Developer-Stack.

-About the solution you are proposing to access sqlite database, let's see if I understand: the idea is to store the db file in my REPO folder (.../edx-platform/ folder where I execute 'vagrant up' to start the virtual machine) instead of storing it in the virtual machine, I'm I right? Does this not cause any problem when Vagrant sync the files in the repo folder with the files in the VM?(duplicate db). I have never work with Vagrant before so excuse me if I'm a bit lost.

-Last question, any file you put into envs/ folder (like the example of "devstack_db_anal.py" you gave me) will be called or should I modify some other file in order to do it?

PS: This topic was duplicate by error (when I posted it, I refresh the group page and I didn't see the topic so I wait ~30min and post it again). Once I could see both topics published I delete one of them (the one you answered me). I'm new to Google Groups and mailing lists so I wonder if there is any way of merge this thread with the other one (https://groups.google.com/forum/#!topic/edx-code/Aa3IFDD3evM)

[Yarko Tymciurak]

Accidentally "replied" instead of "replied-all" - Javier mentioned this would be helpful to others, so here it is.

---------- Forwarded message ----------
From: Yarko Tymciurak <yar...@gmail.com>
Date: Tue, Jan 7, 2014 at 4:20 PM
Subject: Re: [edx-code] Accessing edX datbases from external Django project
To: Javier Santofimia Ruiz <javi...@gmail.com>

On Tue, Jan 7, 2014 at 2:46 PM, Javier Santofimia Ruiz <javi...@gmail.com> wrote:

Hi! Thanks for your answer!
 
-About the db name, yes I had an older version of the platform installed (I installed it 2 months ago). I've just made a fresh install with the method described in https://github.com/edx/configuration/wiki/edX-Developer-Stack.

-About the solution you are proposing to access sqlite database, let's see if I understand: the idea is to store the db file in my REPO folder (.../edx-platform/ folder where I execute 'vagrant up' to start the virtual machine)

well - if you mean "where you execute vagrant up"  according to the current install directions you describe above, - yes.

 

instead of storing it in the virtual machine, I'm I right? Does this not cause any problem when Vagrant sync the files in the repo folder with the files in the VM?(duplicate db). I have never work with Vagrant before so excuse me if I'm a bit lost.

There is no "sync" - the edx-platform and cs_comment_service and ora git repositories exist on your workstation (not your VM).

In the devstack, the VM gets a "view" into your host machine through NFS mounted directies.   To convince yourself of this - bring up your VM ("vagrant up")  and then "vagrant ssh" and "sudo su edxapp" - you will be in your edx-platform directory - it is _the one from your host_.  To convince yourself that it is only a view, here are two things to do:   from VM:  "mount" - and notice where the current directory is mounted to (you will see an IP address - this is your host;  this is the [N]etwork [F]ile [S]ystem mount.   To further convince yourself,  from another terminal on your host, go to this same directory  (in that same REPO folder,  in .../edx-platform)   and make some simple file, maybe:   "$ ehco "hello to me!" > hello.text

Then, from your VM terminal,  ls - you should see "hello.txt"  there.    "$ cat hello.txt"  should show what you just created.   From you VM, then try something like "$ echo "hello from VM!" >> hello.txt"    and then look at that file from your host workstation;  you should see the modification there.

Now, you should understand why the devstack is this way - so you can do development on your host, with familiar tools, while running it on "a machine close to the deployment machine".   This is very cool!

 

-Last question, any file you put into envs/ folder (like the example of "devstack_db_anal.py" you gave me) will be called or should I modify some other file in order to do it?

Look at  "Using the edX devstack" in https://github.com/edx/configuration/wiki/edX-Developer-Stack

The line shown there (for example, in lms) specifies the environment to use for this development instance, for example it shows the default:

• ./manage.py lms runserver --settings=devstack 0.0.0.0:8000

You would change this like so (in fact, you could very easily try different settings by making a few settings files)

 

./manage.py lms runserver --settings=devstack_db_anal 0.0.0.0:8000

Does this help?

[Yarko Tymciurak]

forgot to "reply-all" on this;  Javier requested I post to group also.

---------- Forwarded message ----------
From: Yarko Tymciurak <yar...@gmail.com>
Date: Tue, Jan 7, 2014 at 4:23 PM
Subject: Re: [edx-code] Accessing edX datbases from external Django project
To: Javier Santofimia Ruiz <javi...@gmail.com>

On Tue, Jan 7, 2014 at 4:20 PM, Yarko Tymciurak <yar...@gmail.com> wrote:

On Tue, Jan 7, 2014 at 2:46 PM, Javier Santofimia Ruiz <javi...@gmail.com> wrote:

Hi! Thanks for your answer!
 
-About the db name, yes I had an older version of the platform installed (I installed it 2 months ago). I've just made a fresh install with the method described in https://github.com/edx/configuration/wiki/edX-Developer-Stack.

-About the solution you are proposing to access sqlite database, let's see if I understand: the idea is to store the db file in my REPO folder (.../edx-platform/ folder where I execute 'vagrant up' to start the virtual machine)

well - if you mean "where you execute vagrant up"  according to the current install directions you describe above, - yes.

 

instead of storing it in the virtual machine, I'm I right? Does this not cause any problem when Vagrant sync the files in the repo folder with the files in the VM?(duplicate db). I have never work with Vagrant before so excuse me if I'm a bit lost.

There is no "sync" - the edx-platform and cs_comment_service and ora git repositories exist on your workstation (not your VM).

In the devstack, the VM gets a "view" into your host machine through NFS mounted directies.   To convince yourself of this - bring up your VM ("vagrant up")  and then "vagrant ssh" and "sudo su edxapp" - you will be in your edx-platform directory - it is _the one from your host_.  To convince yourself that it is only a view, here are two things to do:   from VM:  "mount" - and notice where the current directory is mounted to (you will see an IP address - this is your host;  this is the [N]etwork [F]ile [S]ystem mount.   To further convince yourself,  from another terminal on your host, go to this same directory  (in that same REPO folder,  in .../edx-platform)   and make some simple file, maybe:   "$ ehco "hello to me!" > hello.text

Then, from your VM terminal,  ls - you should see "hello.txt"  there.    "$ cat hello.txt"  should show what you just created.   From you VM, then try something like "$ echo "hello from VM!" >> hello.txt"    and then look at that file from your host workstation;  you should see the modification there.

Now, you should understand why the devstack is this way - so you can do development on your host, with familiar tools, while running it on "a machine close to the deployment machine".   This is very cool!

-Last question, any file you put into envs/ folder (like the example of "devstack_db_anal.py" you gave me) will be called or should I modify some other file in order to do it?

Look at  "Using the edX devstack" in https://github.com/edx/configuration/wiki/edX-Developer-Stack

The line shown there (for example, in lms) specifies the environment to use for this development instance, for example it shows the default:

• ./manage.py lms runserver --settings=devstack 0.0.0.0:8000

You would change this like so (in fact, you could very easily try different settings by making a few settings files)

 

./manage.py lms runserver --settings=devstack_db_anal 0.0.0.0:8000

For this to work, be sure that (in this example) your file includes a line  "from .devstack import *"  - just as devstack.py, and aws.py do.

An alternative to this is to look at devstack.py - at the end it tries to import from a "private.py" if one exists.   You could do "fast tests" by making a private.py.   In this case, you would instantiate the lms server as shown in the wiki (you would not need to change the settings).

 

Does this help?