Announcing Mayan EDMS NG version 2.8

[lonevi...@gmail.com]

Hi all,

We love Mayan EDMS and have come to depend on it for our daily work. Like many of you we are worried about the effects hurricane Maria has had on the development of the project. So we decided to do something about it.

We gained good knowledge about the internals of Mayan EDMS from customizing it for our own use. Looking at the code for the next version we could see that much work had already been done. We were confident we could finish the remainder tasks to be able to take the code to a point where a version 2.8 would be possible.

We contacted Roberto with our idea. While his communication options are limited, he nonetheless provided us with some great guidelines. We don't pretend our fork to replace Mayan EDMS so we are naming it Mayan EDMS NG (Next generation). To make things easier we continued the version numbering. 

So, we are proud to release our work as Mayan EDMS NG version 2.8.

Our forks is just a continuation of the work already started by Roberto for version 2.8. Therefore it is 100% compatible with any existing data. 

Our work focused on the following areas:

* Finishing the API test refactor. The purpose of this refactor is to test each API function for permission and access failure and success. The API tests we also updated to conform with the new API Test class interface which reduces a lot of boilerplate code. There are now a minimal of two tests for each API function.

* Some minor permission changes were made when obviously needed such as the Workflow Create permission not having any effect.

* Roberto had already backported the notifications feature to the "next" branch and our work just focused on giving the final polish to this feature by removing duplication.

* Some user interface fixes and updated we added. These were inspired by discussion on the mailing list and from looking at the Open Paperless fork by Tina Zhou.

* Introduction of the MERC (Mayan EDMS Request for Comment) as a means of documenting the internals of the project, proposing features, changes to the code as well as to the related process of the project. This type of documentation mechanism has been very successful for other things like the Internet (RFCs), Python (PEPs) and Django (DEPs) and we are sure it will also help Mayan EDMS a lot.

The repository for this fork is https://gitlab.com/Michael.Price/mayan-edms. From there you can examine the changes that we made to get the code to release level. Our work is available from PyPI as mayan-edms-ng.

We hope you enjoy it, thank you.

Michael Price, internals and process

Eric Riggs, user interface

[RW Shore]

re version 2.8: do you plan to have a Docker container as well? (hint hint :-)

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lonevi...@gmail.com]

Yes we do :) But are trying to push some improvements before releasing it. Eric is working on modernizing the existing Docker image. He is also trying to merge it with the ARM Docker image to provide a single multi architecture image that would work the same on a PC and on a Raspberry Pi. We are also experimenting with Alpine Linux to reduce the image size. Once finished we will announce it on the list.

To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+...@googlegroups.com.

[ericr...@gmail.com]

I managed to get Mayan running using Alpine Linux but the final image size was almost the same using Ubuntu. There were too many path and configuration file changes and while Alpine runs well it is not LSB compliant. I don't think it is a fit for something like as complex as Mayan and needs to run reliably in business settings. Alpine was a bit faster but is missing too many things like language files and fonts (our multi language Office test document doesn't render using Alpine). There are problems with Ubuntu too. Ubuntu 16.04, the recommended version for Mayan, is rather old and missing a recent kernel. Moving to a newer version breaks too many things. Ubuntu is not careful when it comes to backwards compatibility. RedHat, Fedora and CentOS were a disaster. Many basic libraries are missing and the official recommendation is to use third party repositories. No thanks. I'm going to try Debian next, which is very secure but also very stable in terms of release continuity (as opposed to Ubuntu).

Docker adding support for multi architecture images for many of its base images and while not yet official they are very stable and work very well. The Docker image for Raspberry Pi and for the Odroid deployed perfectly from the same Dockerfile file used to deploy for AMD64. However, given that the power and RAM of the ARM boards is considerably less than that of a desktop PC, the Dockerfile for these board will still need to be published as a separate repository. The main problem is that launching 4 workers (fast, medium, slow, OCR) works on a PC, on an ARM board, it consumes all available RAM (1 or 2 GB), which slows the board to a crawl.

We need a way to detect the platform for which the image is being generated and do a conditional branch to select a different template for supervisor for example. The official Postgres image does this but I have yet to understand how it does it.

Another suggestion I have is that we create several versions of the same release. Not really versions but "flavors". For example:

Mayan EDMS mini - For ARM boards, disables file signatures, workflows and OCR. Doesn't include LibreOffice or Tesseract.

Mayan EDMS user or standard - For AMD/Intel servers, disables file signatures and workflows.

Mayan EDMS commercial or enterprise - The current version of the Docker image.

Implementing these is easy. An environment variable is passed to the docker build command specifying the flavor of the image that triggers conditional code in the Docker file.

The current Docker repository URL for the Mayan EDMS image needs to be updated. Currently it is mayanedms/mayanedms. This doesn't allow for other flavors of the image.

A better scheme would be:

mayanedms/standard:2.8

mayanedms/mini:2.8

mayanedms/arm64:2.8 

and so on. It allows for different flavors and version of the product.

[RW Shore]

Good. So far the existing docker image is working fine for me, though I need to figure out how to install the EXIF processing extensions. One request I would make, though, and that's to get rid of the named volume, for the following reasons.

I run mayan-edms in a docker swarm, with glusterfs-based shared disks across the three hosts involved. This means that if/as the service migrates from host to host, the service sees the same data files, independent of the host it's running on. Having a named volume in the mayan-edms image means I need to add a couple of steps to the build process:

1. I need to install a special volume driver, local-persist, so I can re-direct the external definition of the volume out of /var/lib/docker/volumes.

2. I need to define the volume on each of the hosts in the swarm (a volume definition is always local).

By making the external instantiation of /var/lib/mayan a normal mount rather than a volume mount, I can more easily add hosts to the swarm without having to remember to define the mayan_data volume, and I can use bind mounts in the service definition rather than volume mounts, avoiding the need for the local-persist driver.

To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+unsubscribe@googlegroups.com.

[Mathias Behrle]

* ericr...@gmail.com: " Re: [Mayan EDMS: 2288] Announcing Mayan EDMS NG
version 2.8" (Wed, 28 Feb 2018 17:49:27 -0800 (PST)):

> I managed to get Mayan running using Alpine Linux but the final image size
> was almost the same using Ubuntu. There were too many path and
> configuration file changes and while Alpine runs well it is not LSB
> compliant. I don't think it is a fit for something like as complex as Mayan
> and needs to run reliably in business settings. Alpine was a bit faster but
> is missing too many things like language files and fonts (our multi
> language Office test document doesn't render using Alpine). There are
> problems with Ubuntu too. Ubuntu 16.04, the recommended version for Mayan,
> is rather old and missing a recent kernel. Moving to a newer version breaks
> too many things. Ubuntu is not careful when it comes to backwards
> compatibility. RedHat, Fedora and CentOS were a disaster. Many basic
> libraries are missing and the official recommendation is to use third party
> repositories. No thanks. I'm going to try Debian next, which is very secure
> but also very stable in terms of release continuity (as opposed to Ubuntu).

JFTR if size matters there are also the *-slim flavors of Debian images
removing some files usually not needed inside containers.

https://hub.docker.com/_/debian/



--

Mathias Behrle
PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6
AC29 7E5C 46B9 D0B6 1C71 7681 D6D0 9BE4 8405 BBF6

[ericr...@gmail.com]

The problem is Docker. The named volumes initialize in a different way, Mayan relies on that behavior for writing the initial setting files. I see the merit in changing that but it requires changing the way Mayan itself works. For the moment we are trying to keep the major functionality intact and focusing on usability, tests, minor features and bugs. 

We are discussing the possibility of another fork. Mayan EDMS NG will remain 100% compatible with Mayan EDMS. The new fork will deviate and will be the place where we test all the ideas and changes we would like to see in Mayan on the long run. Stuff like changing the initialization and make it work with standard Docker volumes. Things that work well and make the product better will be ported from the new fork back to Mayan EDMS NG and eventually to Mayan EDMS. We want to call this something different to make it clear that it is not completely the same as Mayan. Any idea for the name is welcome.

[ericr...@gmail.com]

Thanks, I'm trying out Debian as the base image and liking it so far.

[RW Shore]

Actually, the whole volume stuff should be taken as a low-priority suggestion. I've got my service build working now, and was even able to add a new analyzer function (from GIT repo  https://gitlab.com/startmat/document_analyzer.git) to an extension of the docker image. I'd rather see a focus on cleanup and new features, rather than on my obscure docker needs.

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+unsubscribe@googlegroups.com.

[Jos vd Snepscheut]

Hello,

I 'm trying to run Mayam EDMS NG 3.0.2 (and 3.0.1) both by using the docker image and by buildinging it according your instructions.

But unfortunatly all my containers keeps restarting.

What am I doing wrong?

Where can i find help?

Thanks,

Jos.

Op dinsdag 27 februari 2018 11:40:20 UTC+1 schreef Michael Price:

[Michael Price]

Do a 

    docker logs <your image name>

it should say why it is restarting.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Mayan EDMS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mayan-edms/Fx8rz4kxPCw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mayan-edms+unsubscribe@googlegroups.com.

[Michael Price]

Found these, from the master himself :)

Packaging Django projects for PyPI by Roberto Rosario
https://www.youtube.com/watch?v=bkeIE90hl3Q

Roberto Rosario - Dockerizing Django projects

https://www.youtube.com/watch?v=A3zKZmB9l64

On Sun, Apr 1, 2018 at 5:07 PM, <ericr...@gmail.com> wrote:

Sure thing. Still the named volume issue is like a thorn. I just want to fix it but I still don't get a few things about the way Mayan is packaged for Docker. 

To unsubscribe from this group and stop receiving emails from it, send an email to mayan-edms+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Mayan EDMS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mayan-edms/Fx8rz4kxPCw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mayan-edms+unsubscribe@googlegroups.com.

[Jos vd Snepscheut]

Hello again,

Thanks for your help. 

After I made a backup I removed the database and than the problem of restarting was gone. I think there is a problem with the upgrade from 2.8 to 3.0.2 

Unfortunately I don't know how to solve that. 

Greetings Jos vd Snepscheut. 

Op zo 1 apr. 2018 23:19 schreef <ericr...@gmail.com>:

From what I've seen it is usually a problem connecting to the database. Be it hosted or in another docker image. Make sure you specify the database driver, the host, user, database name and password correctly. Most of these have to be specified for the database container AND the Mayan container.

To unsubscribe from this group and all its topics, send an email to mayan-edms+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to a topic in the Google Groups "Mayan EDMS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mayan-edms/Fx8rz4kxPCw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mayan-edms+...@googlegroups.com.