Arches 5.0 Docker setup problems

David Osborne

unread,

Mar 3, 2020, 6:13:03 PM3/3/20

to Arches Project

I need a new Arches installation to develop some stuff for a conference presentation, so I thought I'd take the opportunity to not only use the latest version but also run it under Docker.

Following the instructions at https://github.com/archesproject/arches/blob/master/docker/Readme.md, I got as far as Quick Start step 6, docker-compose up, but there were problems creating the database for my archesdemo project.

I've uploaded my docker-compose.yml file and the results of running docker-compose up into a gist at https://gist.github.com/daosborne/cd501c56dc117e066b4f118f15af7e57 to show the problem. Is it obvious what I've missed or is wrong?

I'm running Docker Desktop 2.2.0.3 on MacOS 10.14.6 Mojave.

Thanks!

David

Ben O'Steen

unread,

Mar 3, 2020, 7:33:38 PM3/3/20

to David Osborne, Arches Project

Hi,

I think the key error is on line 63 of the gist you posted. For some reason, it is interpreting file bind mount as a directory and not as a file. If I run the default docker compose file here, and start up the db container, this is the log output I get:

$ docker logs db
Add rule to pg_hba: 0.0.0.0/0
Add rule to pg_hba: replication replicator
Setup master database
2020-03-04 00:28:12.848 UTC [25] LOG: starting PostgreSQL 12.1 (Debian 12.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-04 00:28:12.849 UTC [25] LOG: listening on IPv4 address "127.0.0.1", port 5432
2020-03-04 00:28:12.855 UTC [25] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-04 00:28:12.928 UTC [36] LOG: database system was shut down at 2019-12-03 16:35:30 UTC
2020-03-04 00:28:12.931 UTC [37] postgres@postgres FATAL: the database system is starting up
psql: error: could not connect to server: FATAL: the database system is starting up
2020-03-04 00:28:12.948 UTC [25] LOG: database system is ready to accept connections
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)

postgres ready
Setup postgres User:Password
Creating superuser postgres
ALTER ROLE
Creating replication user replicator
CREATE ROLE
postgres db already exists
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)

/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
CREATE DATABASE
UPDATE 1
You are now connected to database "template_postgis" as user "postgres".
CREATE EXTENSION
GRANT
GRANT
GRANT
CREATE EXTENSION

2020-03-04 00:28:16.764 UTC [25] LOG: received smart shutdown request
2020-03-04 00:28:16.783 UTC [25] LOG: background worker "logical replication launcher" (PID 44) exited with exit code 1
2020-03-04 00:28:16.791 UTC [39] LOG: shutting down
2020-03-04 00:28:16.866 UTC [25] LOG: database system is shut down
Postgres initialisation process completed .... restarting in foreground

Why it is different for you I cannot tell right now. I'm using docker-compose version 1.24.1 (docker engine 19.03.5).

Ben

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesprojec...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups "Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archesprojec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/5f42603a-ca04-4b8f-8e90-92af93d20b40%40googlegroups.com.

David Osborne

unread,

Mar 5, 2020, 10:43:53 AM3/5/20

to Arches Project

Thanks, Ben: your comment helped me make some progress (to another error ;-)

The problem is that the Readme.md in the docker directory of the Github archesproject/arches repo (file last revised 17 months ago!) is inaccurate or vague when it says "Create a new folder for your custom Arches code" without specifying where. I created one alongside my cloned arches repo, with the result that the path to /docker-entrypoint-initdb.d/init.sql did not exist, so Docker created directories instead.

Having located the arches/install/init-unix.sql file referred to in the db service, I worked out that it's easiest to run docker-compose in the cloned arches directory itself, so I edited the docker-compose.yml file, and that file and the resulting output from docker-compose is in https://gist.github.com/daosborne/bc83812e33d9396d679af54f31565895 (with the lines from couchdb and elasticsearch edited out from the docker-compose log as not relevant).

It gets further but has some problems (partly because I want the default graphs and concepts installed):

error setting up db (traceback starting line 133)
database tables graphs, ontologies, concepts, d_value_types don't exist (lines 222--648)
db connection error during migrations (line 734)
db column nodes.fieldname doesn't exist (lines 1123, 1185)

I had also tried setting a project name, as suggested in the Readme but abandoned that because the directory was missing from /web_root (and produces a spurious error message about the project name clashing with a Python module).

I hoped that using Docker would be a quick and easy way of getting a new Arches installation running but it seems it still has some rough edges. As I need to get some development done to make some slides for a conference paper, I think I'll go back to the tried and tested method of setting up a Vagrant VM!

Cheers

David

-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to arches...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en

---
You received this message because you are subscribed to the Google Groups "Arches Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to arches...@googlegroups.com.

Ben O'Steen

unread,

Mar 5, 2020, 6:25:52 PM3/5/20

to David Osborne, Arches Project

Im going to guess that the elasticsearch container is not ready and the migration script fails when it attempts to run the first time through and Arches ends up in a bad state which it generally can never recover from.

docker-compose down -v

This will kill and delete all the badly formed docker containers created by a failed initial run.

docker-compose up --no-start

This will initialize the containers and network for the stack.

docker-compose start elasticsearch db

This will start the elasticsearch instance and the database.

docker logs -f elasticsearch

This will let you watch what it is doing. Ctrl-c to get back to normal. I've found that if you run antivirus checkers and have some network slowness, it can take a while (30s to 1min) for elasticsearch to actually be ready. It sadly lies on its health check command and reports "green" while it is still doing its random clustering things.

docker-compose up

After elasticsearch has finished doing whatever it does on start up that takes so long, run this and watch to make sure there are no connection errors.

Ben

-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesprojec...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en

---
You received this message because you are subscribed to the Google Groups "Arches Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to archesprojec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/7faf5315-c7ae-4aa1-881c-c21378266483%40googlegroups.com.

David Osborne

unread,

Mar 7, 2020, 6:13:44 PM3/7/20

to Arches Project

Hi Ben

Thanks, that was it: starting the elasticsearch and db containers first was the solution.

My docker-compose file asked for the default resource models and concepts. Although the concepts loaded ok, the branches and models didn't, with a missing ontology, although I can't see how to enable whatever's missing:

arches           | Running: python manage.py packages -o import_graphs
arches           | operation: import_graphs
arches           | /web_root/arches/arches/db/graphs/branches/Information Resource Copyright.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Event Phase Type.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Description.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Place 1.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Resource Type Classification.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Publication Event.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/External Identifier.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Heritage Resource Group Phase Type.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Appellation.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Modification Event.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Heritage Resource Phase Type.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Component.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Settings System Defaults.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Evaluation.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Activity Phase.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Place 3.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Resource Update Event.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Keyword.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Beginning of Existence.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Settings Map.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Actor Phase.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Right.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Title.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Information Carrier.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Information Resource Creation Event.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Settings Saved Searches.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Settings Time Search.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Language.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/End of Existence.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Settings Basic Search.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Existence.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Temporal Coverage.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Place 2.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Measurement.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Name.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/branches/Condition Assessment.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/resource_models/Heritage Resource Model.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/resource_models/Activity Resource Model.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/resource_models/Heritage Resource Group Resource Model.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/resource_models/Historical Event Resource Model.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/resource_models/Actor Resource Model.json
arches           | Ontology matching query does not exist.
arches           | /web_root/arches/arches/db/graphs/resource_models/Information Resource Model.json
arches           | Ontology matching query does not exist.

David

To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/7faf5315-c7ae-4aa1-881c-c21378266483%40googlegroups.com.

Dennis Wuthrich

unread,

Mar 7, 2020, 6:28:31 PM3/7/20

to Arches Project

Hi David,

In Arches 5.0, ontologies are loaded as part of a package. See https://github.com/archesproject/arches-her/tree/master/arches_her/pkg for an example of how ontologies are defined. These are automatically loaded as part of the package load command.