Install guide for people who don't really know what they are doing

[Jonathon]

Written by someone who doesn't really know what he's doing either.

This install guide steps through how I got Mayan version 1.1 up and running on a raspberry Pi to act as a home document management server.
The Raspberry Pi is a reasonably low powered computing system, so I have tried to keep things lean by using SQLite and Nginx and not installed Libreoffice (as I will only be storing scanned documents).

Hardware
Mayan EDMS works on the Pi B, but for only a little bit more money the Pi 2 works much better.
Also a class 10 SD card is important to speed up those IO cycles.
I installed Raspbian and stripped out as much of the Xorg stuff as I could by following this stackexchange thread: http://raspberrypi.stackexchange.com/questions/4745/how-to-uninstall-x-server-and-desktop-manager-when-running-as-headless-server

Software
These instructions are based on the following articles that were invaluable:

• http://michal.karzynski.pl/blog/2013/06/09/django-nginx-gunicorn-virtualenv-supervisor/
• http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/
• https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
• https://raymii.org/s/tutorials/HTTP_Strict_Transport_Security_for_Apache_NGINX_and_Lighttpd.html

Set up system user

$ sudo groupadd --system mayan
$ sudo useradd --system mayan --gid mayan --shell /bin/bash --create-home mayan
$ sudo mkdir /home/mayan

It is probably better to use /opt/mayan or /opt/www as the home directory, but I don't know if it make much difference.

Install dependancies

$ sudo apt-get install libjpeg-dev libmagic1 libpng-dev libtiff-dev gcc ghostscript gpgv \
    python-dev python-virtualenv tesseract-ocr unpaper poppler-utils -y
$ sudo apt-get install graphicsmagick nginx openssl supervisor redis-server

Install Mayan EDMS application
Set up a second session so that you can switch back & forth from the mayan user and your normal user with sudo rights.

$ sudo su - mayan
mayan@edms:~$ virtualenv venv
mayan@edms:~$ source venv/bin/activate
(venv)mayan@host ~ $ pip install mayan-edms
(venv)mayan@host ~ $ mayan-edms.py initialsetup
(venv)mayan@host ~ $ mayan-edms.py collectstatic

I got errors using the built-in Python image backend, so we will enable graphicsmagick:

(venv)mayan@host ~ $ vi venv/lib/python2.7/site-packages/mayan/settings/production.py

Add the following lines:

CONVERTER_GRAPHICS_BACKEND = 'converter.backends.graphicsmagick.GraphicsMagick'
CONVERTER_GM_SETTINGS = '-limit files 1 -limit memory 512MB -limit map 1GB -density 200'

Test that it works:

(venv)mayan@host ~ $ mayan-edms.py runserver --settings=mayan.settings.production 0.0.0.0:8000

Point your web browser to the Raspberry Pi on port 8000.
Press Control-c on the terminal windows to quit.

Install Gunucorn

(venv)mayan@host ~ $ pip install gunicorn
(venv)mayan@host ~ $ gunicorn mayan.wsgi:application --bind 0.0.0.0:8080

Test that it works by pointing your browser to the Raspberry Pi on port 8080.

We can set up a start script for gunicorn:

(venv)mayan@host ~ $ vi venv/bin/gunicorn_start

Enter the following lines:

#!/bin/bash

DJANGODIR=/home/mayan/venv/lib
SOCKFILE=/home/mayan/venv/run/gunicorn.sock
USER=mayan
GROUP=mayan
NUM_WORKERS=5  #how many worker processes should Gunicorn spawn, 2x CPU + 1
DJANGO_SETTINGS_MODULE=mayan.settings.production
DJANGO_WSGI_MODULE=mayan.wsgi
LOGLEVEL=debug #options are: debug, info, warning, error, critical

echo "Starting $NAME as `whoami`"
cd $DJANGODIR
source ../bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE

# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

exec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
--workers $NUM_WORKERS \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE \
--log-level=$LOGLEVEL \
--log-file=-

Set permissions and create logging directory:

(venv)mayan@host ~ $ chmod 744 venv/bin/gunicorn_start
(venv)mayan@host ~ $ mkdir venv/logs
(venv)mayan@host ~ $ touch venv/logs/gunicorn_supervisor.log

Switch back to your normal user and set up supervisord to start up the gunicorn process:

$ sudo vi /etc/supervisor/conf.d/mayan.conf

Enter the following:

[program:mayan]
command = /home/mayan/venv/bin/gunicorn_start
user = mayan
stdout_logfile = /home/mayan/venv/logs/gunicorn_supervisor.log
redirect_stderr = true
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8 ; Set UTF-8 as default encoding

Update supervisord:

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl status

Set up Nginx
We will implement some reasonable security (I hope) by enabling SSL with strong ciphers and strict transport security.
Note that the cipher suite that is chosen here requires modern browsers. If you are using on old version of IE then it really is time to upgrade.

First up create a certificate and a strong diffie-helman key:

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/mayan.key -out /etc/nginx/mayan.crt
$ sudo chmod 400 /etc/nginx/mayan.*
$ sudo openssl dhparam -out /etc/nginx/mayan.pem 4096
$ sudo chmod 400 /etc/nginx/mayan.pem

Note that the openssl dhparm comand takes a long time on the Raspberry Pi - it's best to leave it running over night.

Now configure Nginx:

$ sudo vi /etc/nginx/sites-available/mayan

Enter the following:

upstream mayan_app_server {
    server unix:/home/mayan/venv/run/gunicorn.sock fail_timeout=0;
}

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {
    listen 443;
    ssl on;
    server_name mayan;
    ssl_certificate /etc/nginx/mayan.crt;
    ssl_certificate_key /etc/nginx/mayan.key;
    ssl_dhparam /etc/nginx/mayan.pem;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'AES128+EECDH:AES128+EDH';
    ssl_prefer_server_ciphers on;
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    client_max_body_size 4G;
   
    location /static/ {
        alias /home/mayan/venv/local/lib/python2.7/site-packages/mayan/media/static/;
    }
    location /media/ {
        alias /home/mayan/venv/local/lib/python2.7/site-packages/mayan/media/;
    }
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        if (!-f $request_filename) {
            proxy_pass http://mayan_app_server;
            break;
        }
    }
}

Link the config to the enabled sites and restart Nginx:

$ sudo ln -s /etc/nginx/sites-available/mayan /etc/nginx/sites-enabled/mayan
$ sudo rm /etc/nginx/sites-enabled/default
$ sudo /etc/init.d/nginx restart

You can now test by pointing browser at the RPi on port 80 and it should redirect to port 443.

At this point the Mayan EDMS system is usable and you may be happy to leave it there. However you will probably get page timeouts occurring when doing OCR as it is a CPU intensive operation.
To fix this we will configure celery and redis to allow tasks to be handed off to workers without tying up the main user interface process.

Configure Celery & Redis
Switch back to the mayan user and install celery:

(venv)mayan@host ~ $ pip install celery[redis]

Switch to your normal user and set up supervisord to start the celery workers:

$ sudo vi /etc/supervisor/conf.d/mayan-celery.conf

Enter the following:

[program:converter]
command=/home/mayan/venv/bin/celery -A mayan worker -l DEBUG -n worker1.%%h -Ofair -Q converter
user=mayan
numprocs=1
stdout_logfile=/home/mayan/venv/logs/celery-worker.log
stderr_logfile=/home/mayan/venv/logs/celery-worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=998

[program:ocr]
command=/home/mayan/venv/bin/celery -A mayan worker -l DEBUG -n worker2.%%h -Ofair -Q ocr
user=mayan
numprocs=1
stdout_logfile=/home/mayan/venv/logs/celery-worker.log
stderr_logfile=/home/mayan/venv/logs/celery-worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=998

[program:uploads]
command=/home/mayan/venv/bin/celery -A mayan worker -l DEBUG -n worker3.%%h -Ofair -Q uploads,checkouts
user=mayan
numprocs=1
stdout_logfile=/home/mayan/venv/logs/celery-worker.log
stderr_logfile=/home/mayan/venv/logs/celery-worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=998

[group:mayan-celery]
programs=converter,ocr,uploads

This seems inelegant to me - there must be a better way to do this.

Update supervisord:

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl status

Switch back to the mayan user and configure the Mayan EDMS application to use redis:

(venv)mayan@host ~ $ vi venv/lib/python2.7/site-packages/mayan/settings/production.py  

Add these lines to the settings file:

BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
CELERY_ALWAYS_EAGER = False

Restart the gunicorn process to pick up the new settings (as your normal user):

$ sudo supervisorctl restart mayan

Now the user interface should be more responsive and you should be able to upload multiple documents at a time.

You can check the logs to see how things are going:

$ tail -f /home/mayan/venv/logs/celery-worker.log
$ tail -f /home/mayan/venv/logs/gunicorn_supervisor.log

Post-install logging tweaks
Once things are going the way you want them, it is a good idea to edit the config files and set the logging levels to 'error' to prevent logs filling up your SD card:

• /home/mayan/venv/bin/gunicorn_start
• /etc/supervisor/conf.d/mayan-celery.conf

Also add these lines to /etc/nginx/sites-available/mayan:

error_log /var/log/nginx/error.log error;
access_log off;

You could also add the logs in /home/mayan/venv/logs to logrotate.

[Pierpaolo Baldan]

you have done a beautiful job, thanks

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roberto Rosario]

Wow! That is impressive! Thanks for sharing this guide!

...

[Cailen McQuattie]

This was extremely helpful, thank you so much

...

[Gabriel Diteko]

Brilliant! Thank you so much! You are a life saver! Though I'm using apache2 for deployment, these settings worked wonders for me.

[Mark Phillips]

This is really great!

Is your setup geared towards development or production? Your setup appears to be using SQLite as the database, and I thought that was not recommended for a production installation.

Mark

On Thu, Jun 11, 2015 at 6:23 AM, Gabriel Diteko <royalp...@gmail.com> wrote:

Brilliant! Thank you so much! You are a life saver! Though I'm using apache2 for deployment, these settings worked wonders for me.

--

[Jonathon]

It is a "production" system for my personal use :) The idea was to keep things as lightweight as possible, so I chose to use SQLite.

The only downside that I am aware of with SQLite is that there might be concurrency problems where there are a large number of transactions per second, but for my home document server I am unlikely to reach the limit. The SQLite documentation (http://www.sqlite.org/whentouse.html) says that it is ok for websites with up to 100,000 hits per day - although maybe not when using a Raspberry Pi and an SD card for storage.

Jonathon.

[Scott hodgson]

Thanks.  Great job. I would ask that people copy the code by hand as I ran into a few problems with the code when cut and pasted, eg end of lines, too many spaces etc.

I haven't started celery and redis yet, as I want to test it before I implement them.

I would like to know is how to change/specify where the storage is. I have looked for document_storage, but it is currently not in the directory venv/lib/python2.7/site-packages/mayan/media.  Any help?

[Scott hodgson]

Actually I think i just need to mount a file system to the directory document_storage, is that right?

[Jonathon Exley]

Yes, there were a couple of typos in my write-up. Unfortunately there is no wiki to edit for mayan.

I think i forgot to mention that you need to create the mayan/venv/lib/python2.7/site-packages/mayan/media/document_storage directory. Oops.

--