Arches local installation on virtual machine

[couim...@gmail.com]

Hello,

I'm attempting a new setup on a virtual machine, and I have some questions regarding my setup. I think I'm almost there, including the installation of the provided HIP.

Here is some info on my setup.

-Running Ubuntu Linux 14-04 via Parallels for Mac.

-I believe that I successfully installed the pre-requisites.

-I think I was successful in installing PIP and have Python (from what I could find out, Python comes with Ubuntu/Linux?) 

On my first attempt, I had most of it set up, and could see the setup running when looking at the http://localhost:8000/ page. The only thing that was not working was the Map menu, which was blank probably due to the settings in the "settings.py" file, and the search menu was not working, probably due to the Database settings in the "settings.py" file. 

I've since modified the "settings.py" file (mainly for the map settings and obtained the map key for  Bing), and I completely powered down my virtual machine. I'm attempting to re-run my setup to see if my changes to the settings.py file is working, but I'm now having problems re-starting the elasticsearch (with several permission denied messages) and re-staring Arches by running the "python manage.py run server" (receiving a "SyntaxError: invalid syntax").

I'm probably not doing things in the right order when re-starting my setup. The following is the order I tried to follow to re-launch.

1-In terminal, run "source ENV/bin/activate"

2-In 2nd terminal window, start the elastic search (errors such as: "AccessDeniedException" and "Failed to obtain node lock, is the following location writable?")

3-Even though the elastiseach is not working, I attempted to run Arches on the 1st terminal with "python manage.py run server" but receive "SyntaxError: invalid syntax". I also tried to re-run the "python manage.py packages -o install" script/command but receive the same error.

I suspect that I'm not following the correct order, or I need to go further back such as re-installing the HIP package. I'll be regularly shutting down Ubuntu/Linux, and re-starting it.

I'm not at all an expert with all of this, and I don't neccesarly know what all the pieces do (or using the correct terminology), which can be part of the problem when trying to troubleshoot. Any help is appreciated, and I'm sure I'll have more questions once I get my setup figured out.

Thanks,

Christian

[Darcy Christ]

Hi Christian,

Make sure you use the installation script: https://github.com/archesproject/arches/blob/master/arches/install/ubuntu_trusty_setup.sh

It is very important to do everything within the virtual environment, as that makes sure you are working with the right libraries. Python on Linux is usually an older version. You must be using Python 2.7. The same goes for using Postgresql 9.3. These are requirements right now.

Elasticsearch is essential for using arches. Not only does it power the search, but it also handles some caching.

The proper command is "python manage.py runserver <ip address>:<port>"

Hope this helps!

Darcy

[couim...@gmail.com]

Thanks Darcy, 

I'll give this a try this week and will post my progress. I think I'll start from scratch, since I did not install with the ubuntu_trusty_setup.sh. I used the instructions using the link from the Archesproject site.

Christian

[couim...@gmail.com]

Hello,

I've tried to install several times and have gone through the posts and instructions and seem to be stuck with the get-pip step. Here is an overview of my progress.

Step 1:

-Installed virtual machine (Ubuntu on Mac) and allocated 4GB of RAM

Step 2:

Following instructions found on on "https://github.com/archesproject/arches" Installed dependencies: (replaced bitbucket link with github)

wget https://github.com/archesproject/arches/blob/master/arches/install/ubuntu_trusty_setup.sh

chmod +x ubuntu_trusty_setup.sh

./ubuntu_trusty_setup.sh

This took about 5 minutes to install and all dependencies seem to be installed!

Step 3:

The next step was downloading “get-pip.py” 

I then ran “python get-pip.py” but received the following errors or exceptions that appear in red text. The link provided in blue text bellow suggest that I upgrade to python 2.7.9

Collecting pip

/tmp/tmp0SyKHy/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.

/tmp/tmp0SyKHy/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

  Downloading pip-8.1.1-py2.py3-none-any.whl (1.2MB)

    100% |████████████████████████████████| 1.2MB 396kB/s 

Collecting setuptools

  Downloading setuptools-20.9.0-py2.py3-none-any.whl (508kB)

    100% |████████████████████████████████| 512kB 471kB/s 

Collecting wheel

  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)

    100% |████████████████████████████████| 71kB 621kB/s 

Installing collected packages: pip, setuptools, wheel

Exception:

Traceback (most recent call last):

  File "/tmp/tmp0SyKHy/pip.zip/pip/basecommand.py", line 209, in main

    status = self.run(options, args)

  File "/tmp/tmp0SyKHy/pip.zip/pip/commands/install.py", line 317, in run

    prefix=options.prefix_path,

  File "/tmp/tmp0SyKHy/pip.zip/pip/req/req_set.py", line 732, in install

    **kwargs

  File "/tmp/tmp0SyKHy/pip.zip/pip/req/req_install.py", line 835, in install

    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)

  File "/tmp/tmp0SyKHy/pip.zip/pip/req/req_install.py", line 1030, in move_wheel_files

    isolated=self.isolated,

  File "/tmp/tmp0SyKHy/pip.zip/pip/wheel.py", line 344, in move_wheel_files

    clobber(source, lib_dir, True)

  File "/tmp/tmp0SyKHy/pip.zip/pip/wheel.py", line 315, in clobber

    ensure_dir(destdir)

  File "/tmp/tmp0SyKHy/pip.zip/pip/utils/__init__.py", line 83, in ensure_dir

    os.makedirs(path)

  File "/usr/lib/python2.7/os.py", line 157, in makedirs

    mkdir(name, mode)

OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'

I tried to proceed further, but suspect that this error may be causing the problems. 

Any help would be appreciated. 

Thanks,

Christian

[Adam Cox]

Hi Christian, did you try running that same command with sudo privileges? Though it depends on what user account you are logged onto the server with, this is probably necessary.

sudo python get-pip.py

Hopefully that will allow you to install pip, and then you can continue without more errors.

Even though the link recommends that you use Python 2.7.9, be sure that you stick with 2.7.6.  There are some issues with other Arches code if you use a later python version than 2.7.6.

Good luck!

Adam

[couim...@gmail.com]

Thanks Adam, adding sudo helped. 

I was able to install pip (using: sudo apt-get install python-pip), and create the virtual environment. I think I'm now having errors relating to the Arches installation. (I'm almost there!)

Here is the steps I've taken:

sudo pip install virtualenv==1.11.4 (works OK)

Create "Project" directory enter Project directory.

virtualenv ENV (works OK)

source ENV/bin/activate (works OK)

pip install arches (sample of errors throughout instalation as follows:)

sample at start of installation:

    warning: no previously-included files found matching 'arches/logs/*.txt'

    warning: no previously-included files matching '*' found under directory 'tests'

    warning: no previously-included files matching '*' found under directory 'virtualenv'

    warning: no previously-included files matching '*' found under directory 'arches/elasticsearch'

    warning: no previously-included files matching '*' found under directory 'arches/uploadedfiles'

Sample at middle, just before elastic search download:

    compilation terminated.

    

    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

and at end:

    compilation terminated.

    

    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    

    ----------------------------------------

    Cleaning up...

    Command /home/couimet/Projects/ENV/bin/python -c "import setuptools, tokenize;__file__='/home/couimet/Projects/ENV/build/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-pt7G0M-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/couimet/Projects/ENV/include/site/python2.7 failed with error code 1 in /home/couimet/Projects/ENV/build/psycopg2

    Storing debug log for failure in /home/couimet/.pip/pip.log

     30917670  [100.00%]

Successfully installed arches

Cleaning up...

(ENV)couimet@ubuntu:~/Projects$ 

I also tried with sudo pip install arches, but that seems to make it worst.

Thanks again.

Christian

[Adam Cox]

Hi Christian, yes, once you are working in the virtual environment, you shouldn't use sudo anymore to run the python commands.

Can you confirm what version of python you have in your virtual environment?  When you have ENV activated, type

python

and you'll see the working version number.  To exit the python interface in the command line, type

exit()

Your ENV should be running Python 2.7.6.

Adam

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesprojec...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups "Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archesprojec...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[couim...@gmail.com]

Hi Adam,

Just checked and I do have Python 2.7.6. installed.

The Arches installation file (arches-3.1.2.tar.gz) file downloads without issues when running "pip install arches"

Here is the first error that appears bellow. It gives a warning relating to a previously-included file is not found. The install takes less than 20 seconds.

(ENV)couimet@ubuntu:~/Projects$ pip install arches

Downloading/unpacking arches

  Downloading arches-3.1.2.tar.gz (18.9MB): 18.9MB downloaded

  Running setup.py (path:/home/couimet/Projects/ENV/build/arches/setup.py) egg_info for package arches

    

    warning: no previously-included files found matching 'arches/logs/*.txt'

All of the other items (prerequisites and pip) seem to have installed correctly. I tried to reinstall everything with a fresh install of the virtual machine, but have the exact same situation. Could it be that I'm not using an up to date file somewhere.

Thanks again Adam, greatly appreciated. 

Chris

[Alexei Peters]

Hi Chris,

The warning you're seeing isn't an issue.  It's related to our MANIFEST.in file that setup.py uses to build the distribution for pypi.

The build is trying to exclude all *.txt file in the arches/logs directory, but can't find any (which is ok).  Not sure why this shows up as a warning as it shouldn't be an issue at all.

It looks like everything else installed correctly.  If that's the case then Arches is ready to go.  Are you planning on installing the Arches-HIP package?

Cheers,

Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

--

[couim...@gmail.com]

Hi Alexei,

I did try to install the Arches-HIP package, but I receive the following errors. This is why I thought that the Arches installation did not work.

(ENV)couimet@ubuntu:~/Projects$ pip install arches_hip

Downloading/unpacking arches-hip

  Downloading arches_hip-1.0.4.tar.gz (456kB): 456kB downloaded

  Running setup.py (path:/home/couimet/Projects/ENV/build/arches-hip/setup.py) egg_info for package arches-hip

    

    warning: no previously-included files found matching 'arches_hip/logs/*.txt'

    warning: no previously-included files matching '*' found under directory 'tests'

    warning: no previously-included files matching '*' found under directory 'arches_hip/elasticsearch'

    warning: no previously-included files matching '*.log' found anywhere in distribution

    warning: no previously-included files matching '*.zip' found anywhere in distribution

    warning: no previously-included files matching '*.DS_Store' found anywhere in distribution

Requirement already satisfied (use --upgrade to upgrade): arches>=3.1.0 in ./ENV/lib/python2.7/site-packages (from arches-hip)

Installing collected packages: arches-hip

  Running setup.py install for arches-hip

    

    warning: no previously-included files found matching 'arches_hip/logs/*.txt'

    warning: no previously-included files matching '*' found under directory 'tests'

    warning: no previously-included files matching '*' found under directory 'arches_hip/elasticsearch'

    warning: no previously-included files matching '*.log' found anywhere in distribution

    warning: no previously-included files matching '*.zip' found anywhere in distribution

    warning: no previously-included files matching '*.DS_Store' found anywhere in distribution

  Could not find .egg-info directory in install record for arches-hip

Successfully installed arches-hip

Cleaning up...

(ENV)couimet@ubuntu:~/Projects$ 

[Alexei Peters]

I actually think everything installed correctly.  Those are the same type of warnings you had previously.

It actually says "Successfully installed arches-hip" so that's a good thing!

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

[couim...@gmail.com]

Thanks Alexei,

That's good news.

I'm now attempting to install ElasticSearch. I get the following results:

(ENV)couimet@ubuntu:~/Projects/my_hip_app$ python manage.py packages -o setup_elasticsearch

Traceback (most recent call last):

  File "manage.py", line 28, in <module>

    execute_from_command_line(sys.argv)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line

    utility.execute()

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 272, in fetch_command

    klass = load_command_class(app_name, subcommand)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 75, in load_command_class

    module = import_module('%s.management.commands.%s' % (app_name, name))

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module

    __import__(name)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/arches/management/commands/packages.py", line 28, in <module>

    from arches.app.utils.data_management.resources.importer import ResourceLoader

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/arches/app/utils/data_management/resources/importer.py", line 8, in <module>

    from django.contrib.auth.models import User

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/contrib/auth/models.py", line 48, in <module>

    class Permission(models.Model):

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 96, in __new__

    new_class.add_to_class('_meta', Options(meta, **kwargs))

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 264, in add_to_class

    value.contribute_to_class(cls, name)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/models/options.py", line 124, in contribute_to_class

    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/__init__.py", line 34, in __getattr__

    return getattr(connections[DEFAULT_DB_ALIAS], item)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/utils.py", line 198, in __getitem__

    backend = load_backend(db['ENGINE'])

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/utils.py", line 113, in load_backend

    return import_module('%s.base' % backend_name)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module

    __import__(name)

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 1, in <module>

    from django.db.backends.postgresql_psycopg2.base import *

  File "/home/couimet/Projects/ENV/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 25, in <module>

    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named psycopg2

(ENV)couimet@ubuntu:~/Projects/my_hip_app$ 

But after this step, I only have the following directories shown in blue.

Projects
  /ENV (virtual environment where arches_hip gets installed)
  /my_hip_app
    manage.py
    wsgi.py
    README.txt
    /my_hip_app
        settings.py
        setup.py
        urls.py
        /elasticsearch
        /logs
        /media
        /models
        /source_data
        /templates
        /tests
        /views

The /ElasticSearch and /Tests directories are not there. Therefore, I can't start ElasticSearch since it's not found.

Thanks,

Chris

[Adam Cox]

Hi Chris and Alexei, I think the django import error is evidence that the install did not work correctly. Somehow it finished thinking that it had, but some of the pieces did not get fully installed.  To start over in the same virtual environment you can run

pip uninstall arches

I'm surprised that you are getting this error in Ubuntu.  One thing I like to recommend is printing the entire install process output to a text file. Use 

pip install arches > full_output.txt 2>&1

Looking through the resulting "full_output.txt" may give us some clues about what is going on.  However, as Alexei mentioned, there are certain errors or warnings that you can ignore. 

What virtual machine are you using?  I'm surprised that you are finding these errors with Ubuntu.  It make have something to do with the privileges that your user has.

Adam

On Sunday, April 17, 2016 at 11:59:27 AM UTC-6, couim...@gmail.com wrote:

[Adam Cox]

Hi Chris, also just wanted to double check that you have the virtual environment activated when you try the elasticsearch installation...

[couim...@gmail.com]

Hi Adam,

I attached the file relating to the Arches installation (pip install arches > full_output.txt 2>&1).

Here are some of the specs that I'm using:

Parallels for Mac (Virtual machine software)

Ubuntu 14.04 LTS (set up with 4GB Ram)

Yes I believe I have the virtual environment activated when trying to activate the virtual machine. (Env) is the first thing shown in the command line. What is strange after I install the Arches-Hip is that there is no /ElasticSearch or /Tests directory as shown in the example directory structure provided in the instructions (shown bellow). Although if there is a problem with the Arches install, this may be the cause. I may start from scratch tomorrow by reinstalling Ubuntu, the dependencies etc... to see if this works (although I've tried this before).

Projects
  /ENV (virtual environment where arches_hip gets installed)
  /my_hip_app
    manage.py
    wsgi.py
    README.txt
    /my_hip_app
        settings.py
        setup.py
        urls.py
        /elasticsearch
        /logs
        /media
        /models
        /source_data
        /templates
        /tests
        /views

I also had a look at various settings relating to the Ubuntu account. I did not find anything specific relating to the admin rights.

Thanks again Adam for your help.

Chris

---

full_output.txt

[Adam Cox]

Hi Chris, from looking at the output file, it seems like the error comes while trying to install some of the extra python dependencies (pillow, to be exact).  I did a little searching, and it looks like this may be your problem: http://www.cyberciti.biz/faq/debian-ubuntu-linux-python-h-file-not-found-error-solution/

So, try running

sudo apt-get install python-dev

and then making a new virtual environment and installing arches there (or pip uninstalling and arches and then reinstalling it on your existing virtual environment.

Hope that helps,

Adam

On Sunday, April 17, 2016 at 11:59:27 AM UTC-6, couim...@gmail.com wrote: