RHEL (and probably CentOS) Installation Guide

[Rob]

# CRITs Installation on Red Hat Enterprise Linux

# Be sure to reference the official CRITs installation procedures located here: https://github.com/crits/crits

# Notes: This is written for users with some Linux experience. Certain details (like when to "cd" or instructions for where to place files when using "wget") are left out to keep the guide concise.

# Prerequisites (out of scope of this guide).

Install RHEL (I used 6.4 x64 minimal build)

Configure networking

Subscribe the system to Red Hat

# Configure repositories.

yum install wget.x86_64

wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

rpm -ivh epel-release-6-8.noarch.rpm

yum install yum-utils.noarch

yum-config-manager --enable rhel-6-server-optional-rpms

yum clean all

yum makecache

# Install some helpful packages ("Development tools" is probably required).

yum install man.x86_64 mlocate.x86_64

yum groupinstall "Development tools"

# Configure iptables as desired (out of scope of this guide, but note that connections to port 443 and possibly 80 are required).

# Helpful reference for configuring iptables: http://wiki.centos.org/HowTos/Network/IPTables

# RHEL comes with Python 2.6, but CRITs requires Python 2.7. It is ill-advised to upgrade the system's version of Python, so we need to setup a Python 2.7 virtual environment for CRITs to run in.

# Helpful references for this:

# https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4

# https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

# Install dependencies for Python/pip/virtualenv.

yum install zlib-dev openssl-devel sqlite-devel bzip2-devel ncurses.x86_64 ncurses-devel.x86_64 gdbm.x86_64 gdbm-devel.x86_64 readline.x86_64 readline-devel.x86_64

# Install Python 2.7. IMPORTANT: Use "make altinstall" instead of "make install" to avoid replacing the existing Python 2.6 binary.

wget https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz

tar -zxf Python-2.7.8.tgz

./configure --enable-shared LDFLAGS="-Wl,-rpath=/usr/local/lib"

make

make altinstall

# Install pip.

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-5.7.tar.gz

tar -zxf setuptools-5.7.tar.gz

python2.7 setup.py install

curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py | python2.7 -

# Install the virtualenv module and create/activate the Python 2.7 virtual environment for CRITs to use.

pip install virtualenv

virtualenv /usr/CRITsVirtualEnv

source /usr/CRITsVirtualEnv/bin/activate

# Setup the CRITs directory and clone the git repositories.

mkdir /data

mkdir /data/db

cd /data

git clone https://github.com/crits/crits.git

git clone https://github.com/crits/crits_dependencies.git

# Install the dependencies of the CRITs dependencies.

yum install libyaml.x86_64 libyaml-devel.x86_64 ucl.x86_64 ucl-devel.x86_64 libxml2.x86_64 libxml2-devel.x86_64 libxslt.x86_64 libxslt-devel.x86_64

# Fix errors in the CRITs dependency installer script and then run the script.

cd /data/crits_dependencies

sed -ie 's/p7zip-9\.20\.1-2\.el6\.rf\.x86_64\.rpm/p7zip-9\.20\.1-2\.el6\.x86_64\.rpm/' install_dependencies.sh

sed -ie 's/\smatplotlib/ python-matplotlib.x86_64/' install_dependencies.sh

./install_dependencies.sh

# Install other CRITs dependencies that have manual installation procedures.

# M2Crypto.

git clone https://github.com/M2Crypto/M2Crypto

./fedora_setup.sh build

./fedora_setup.sh install

# PIL.

yum install libjpeg.x86_64 libjpeg-devel.x86_64 zlib.x86_64 zlib-devel.x86_64 freetype.x86_64 freetype-devel.x86_64 lcms2-devel.x86_64

pip install PIL --allow-external PIL --allow-unverified PIL

# mod_wsgi

yum install httpd-devel.x86_64

wget http://modwsgi.googlecode.com/files/mod_wsgi-3.4.tar.gz

tar -zxf mod_wsgi-3.4.tar.gz

./configure --with-python=/usr/CRITsVirtualEnv/bin/python

make

make install

# Adjust TCP server parameters.

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

echo "echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse" >> /etc/rc.local

echo "echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle" >> /etc/rc.local

# Setup the MongoDB database.

cd /data/crits/contrib/mongo/UMA/

./mongod_start.sh

# [Optional] Verify MongoDB started successfully.

mongo

quit()

# Configure and start CRITs

adduser crits

usermod -G crits apache

touch /data/crits/logs/crits.log

chmod 664 /data/crits/logs/crits.log

chgrp -R crits /data/crits/logs

cp /data/crits/crits/config/database_example.py /data/crits/crits/config/database.py

# Edit /data/crits/crits/config/database.py as necessary. In particular, set the SECRET_KEY value to the output of the following or manage.py will not work correctly.

python

from django.utils.crypto import get_random_string as grs

print grs(50, 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)')

cd /data/crits

python manage.py create_default_collections

# Make note of the temporary password created in the following step.

python manage.py users -a -u <userid> -f <fname> -l <lname> -e <email>

python manage.py setconfig allowed_hosts "<ip>,<hostname>,<fqdn>,<etc>"

# Configure Apache.

service httpd stop

cd /data/crits/extras

cp rhel_httpd.conf /etc/httpd/conf/httpd.conf

cp rhel_ssl.conf /etc/httpd/conf.d/ssl.conf

# Fix errors in the CRITs Apache configuration files.

sed -ie 's/^WSGIPythonPath\s\/data\/crits$//' /etc/httpd/conf.d/ssl.conf

echo -e WSGIPythonPath /data/crits:/usr/CRITsVirtualEnv/lib/python2.7/site-packages | sudo tee -a /etc/httpd/conf/httpd.conf

# Creating a self-signed certificate is bad, m'kay.

openssl req -new > new.cert.csr

openssl rsa -in privkey.pem -out new.cert.key

openssl x509 -in new.cert.csr -out new.cert.cert -req -signkey new.cert.key -days 3650

cp new.cert.cert /etc/pki/tls/certs/crits.crt

cp new.cert.key /etc/pki/tls/private/crits.plain.key

# Disabling SELinux is bad, m'kay.

setenforce 0

service httpd start

# That should be it. Access CRITs by browsing to https://<IP/hostname>/crits

[Michael Goffin]

This is great! Would you be up for writing this in Markdown or some format that Github will format? I'd be willing to put this up on the wiki as an alternate reference guide people can use. Then it can be iterated over and kept up-to-date from there!

--
You received this message because you are subscribed to the Google Groups "crits-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crits-users...@googlegroups.com.
To post to this group, send email to crits...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crits-users/64af2ca1-6fda-4d5b-9b05-0ef8fac86ad3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob]

Will do. Might take me a day or two but I'll post it here when it's done.

[Rob]

Markdown is attached. Let me know if you have any questions.

[Sam]

I'm trying to use these instructions on a Redhat 6.5 64bit machine on my corporate network, which is fronted by a proxy.  I'm seeing the following errors near the end of the "./install_dependencies.sh" command.  I believe it may have to do with my proxy, https, and pip or some other tool trying to install things?  Any ideas?  I'm at a loss. 

Installed /usr/local/lib/python2.7/site-packages/django_tastypie-0.11.0-py2.7.egg

Processing dependencies for django-tastypie==0.11.0

Searching for python-dateutil>=1.5,!=2.0

Reading https://pypi.python.org/simple/python-dateutil/

Download error on https://pypi.python.org/simple/python-dateutil/: [Errno 110] Connection timed out -- Some packages may not be found!

Couldn't find index page for 'python-dateutil' (maybe misspelled?)

Scanning index of all packages (this may take a while)

Reading https://pypi.python.org/simple/

Download error on https://pypi.python.org/simple/: [Errno 110] Connection timed out -- Some packages may not be found!

No local packages or download links found for python-dateutil>=1.5,!=2.0

error: Could not find suitable distribution for Requirement.parse('python-dateutil>=1.5,!=2.0')

Installing Django Tastypie Mongoengine 0.4.5...

./install_dependencies.sh: line 100: cd: django-tastypie-mongoengine-0.4.5: No such file or directory

Installing MongoEngine 0.8.7...

./install_dependencies.sh: line 102: cd: mongoengine-0.8.7: No such file or directory

Installing ssdeep...

./install_dependencies.sh: line 104: cd: ssdeep-2.11: No such file or directory

./install_dependencies.sh: line 105: cd: pydeep-0.2: No such file or directory

Installing Python magic...

./install_dependencies.sh: line 113: cd: python-magic: No such file or directory

Installing dependencies for Services Framework...

./install_dependencies.sh: line 115: cd: anyjson-0.3.3: No such file or directory

./install_dependencies.sh: line 116: cd: amqp-1.0.6: No such file or directory

./install_dependencies.sh: line 117: cd: billiard-2.7.3.19: No such file or directory

./install_dependencies.sh: line 118: cd: kombu-2.5.4: No such file or directory

./install_dependencies.sh: line 119: cd: celery-3.0.12: No such file or directory

./install_dependencies.sh: line 120: cd: django-celery-3.0.11: No such file or directory

./install_dependencies.sh: line 121: cd: requests-v1.1.0-9: No such file or directory

./install_dependencies.sh: line 122: cd: cybox-2.1.0.5: No such file or directory

./install_dependencies.sh: line 123: cd: stix-1.1.1.0: No such file or directory

Dependency installations complete!

[Sam]

On a Redhat 6.5 64 bit machine, the install_dependencies.sh script won't correctly detect the OS.  Below is the relevant piece of the script that drops into the first if clause on a Redhat 6.5 64 bit machine.  I had to modify the if clause to get the script to continue.

# Using lsb-release because os-release not available on Ubuntu 10.04

if [ -f /etc/lsb-release ]; then

    . /etc/lsb-release

    OS=$DISTRIB_ID

    VER=$DISTRIB_RELEASE

elif [ -f /etc/redhat-release ]; then

    OS=$(cat /etc/redhat-release | sed 's/ Enterprise.*//')

    VER=$(cat /etc/redhat-release | sed 's/.*release //;s/ .*$//')

else

    OS=$(uname -s)

    VER=$(uname -r)

fi

On Tuesday, September 9, 2014 9:16:04 AM UTC-7, Rob wrote:

[S Siu]

I am running into the same issue on Ubuntu 14.04 also behind a corporate proxy, but it appears that my OS is being detected correctly.  Did you do anything else to fix the issue?

I added an echo to the script right after the OS check you mentioned, and this is what it has in the variables.
$ sudo -E ./install_dependencies.sh
Architecture: 64
OS:  Ubuntu
VER: 14.04

Like your post above, below is the error that I am seeing:

Processing django_tastypie_mongoengine-0.4.5-py2.7.egg
removing '/usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg' (and everything under it)
creating /usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg
Extracting django_tastypie_mongoengine-0.4.5-py2.7.egg to /usr/local/lib/python2.7/dist-packages
django-tastypie-mongoengine 0.4.5 is already the active version in easy-install.pth

Installed /usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg
Processing dependencies for django-tastypie-mongoengine==0.4.5
Searching for biplist
Reading https://pypi.python.org/simple/biplist/
Download error on https://pypi.python.org/simple/biplist/: [Errno 110] Connection timed out -- Some packages may not be found!
Couldn't find index page for 'biplist' (maybe misspelled?)

Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [Errno 110] Connection timed out -- Some packages may not be found!

No local packages or download links found for biplist
error: Could not find suitable distribution for Requirement.parse('biplist')
Installing MongoEngine 0.8.7...
./install_dependencies.sh: line 114: cd: mongoengine-0.8.7: No such file or directory
Installing ssdeep...
./install_dependencies.sh: line 116: cd: ssdeep-2.11: No such file or directory
./install_dependencies.sh: line 117: cd: pydeep-0.2: No such file or directory
Installing Python magic...
./install_dependencies.sh: line 125: cd: python-magic: No such file or directory

Installing dependencies for Services Framework...

./install_dependencies.sh: line 127: cd: anyjson-0.3.3: No such file or directory
./install_dependencies.sh: line 128: cd: amqp-1.0.6: No such file or directory
./install_dependencies.sh: line 129: cd: billiard-2.7.3.19: No such file or directory
./install_dependencies.sh: line 130: cd: kombu-2.5.4: No such file or directory
./install_dependencies.sh: line 131: cd: celery-3.0.12: No such file or directory
./install_dependencies.sh: line 132: cd: django-celery-3.0.11: No such file or directory
./install_dependencies.sh: line 133: cd: requests-v1.1.0-9: No such file or directory
./install_dependencies.sh: line 134: cd: cybox-2.1.0.5: No such file or directory
./install_dependencies.sh: line 135: cd: stix-1.1.1.0: No such file or directory
Dependency installations complete!

For further inspection, I went into the django-tastypie-mongoengine directory and ran:

sudo -E python setup.py install

And I got the following information:

Installed /usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg
Processing dependencies for django-tastypie-mongoengine==0.4.5
Searching for biplist
Reading https://pypi.python.org/simple/biplist/
Download error on https://pypi.python.org/simple/biplist/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
Couldn't find index page for 'biplist' (maybe misspelled?)

Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/

Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
No local packages or download links found for biplist
error: Could not find suitable distribution for Requirement.parse('biplist')

For another install I had used: echo insecure >> ~/.curlrc

Do I need to do something similar for pycurl as mentioned here?  I also saw something about an issue with urllib3 in pip, and tried the solution suggested here in /usr/lib/python2.7/dist-packages/requests/adapters.py, but it didn't seem to help.

Any input is greatly appreciated, thanks!


[Cris Rhea]

Just FYI-- this information (along with the page at https://github.com/crits/crits/Wiki/RHEL-Supplemental-Install-Guide)  is VERY out of date.

Being new to CRITs (and not understanding how all the pieces fit together), I spent a couple days trying to follow the outdated info. 

The current (Nov 2016) helper scripts (scripts/bootstrap) have a different approach for python 2.7 (separate install in /usr/local/bin vs virtualenv) which

really messes with getting all this running correctly. (There are other issues, but Python version causes the biggest mess...)

I really appreciate the work Rob did to put this together (Thanks Rob!).

If there is interest, I can provide a similar cookbook for what I had to do to get CRITs working (Nov 2016 CRITs + CentOS/RedHat 6).

[C. L. Martinez]

Yes, please, please. It would be of great help for a some people in this list which we prefer RHEL/CentOS over Ubuntu for production servers :)

--
Greetings,
C. L. Martinez

[Cris Rhea]

OK, I edited my notes into something someone else could possibly read. I'd love it if you could try it and see if it makes sense to you.

If this is useful to the group, I'll also write-up the cookbook for getting many of the crits_services working in this environment.