Installation of Archipelago on RHEL 7.6

[CSHL Library]

Good Afternoon,

I am running Archipelago 8.x-1.0-beta2 on a VM running RHEL 7.6.

My username has been added to the docker group as suggested by the install instructions,

The VM already has a existing instance of Drurpal running, just to make life interesting.

We created a digital object collection and placed a digital object containing a jpg image into that first collection.

In the area expected to show the .jpg, the following error is displayed:

    Unable to open [object Object]: HTTP 0 attempting to load TileSource

In the area I expect to see the thumbnail, I only see the text "Thumbnail".

I also expected there to be a few pre-installed items, but they did not appear.

Any insights on how to get past this?

My main deviation from the install instructions was due to port conflict on minio, so I changed from port 9000 to port 9001.

The server did not have docker installed so I followed the instruction posted at:

https://www.unixarena.com/2018/06/how-to-install-docker-on-redhat-linux-rhel-centos.html/

Docker installation (sudo):

rpm -qa | egrep "mapper|lvm2|device-mapper-persistent-data|yum-utils"

yum install ca-certificates
yum install curl

yum install git

yum install yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum-config-manager --enable rhel-7-server-extras-rpms
yum install docker-ce

yum install docker-compose

systemctl start docker
systemctl enable docker
systemctl status docker

docker run -it hello-world
usermod -aG docker myusername

su - myusername
id -nG
docker run -it hello-world

Archipelago installation (as user):

git clone https://github.com/esmero/archipelago-deployment.git archipelago-deployment
cd archipelago-deployment/
git checkout 8.x-1.0-beta2
cp docker-compose-nginx.yml  docker-compose.yml

vi docker-compose.yml     # Change minio port to 9001 due to a conflict on 9000 port.

ls -l persistent

mkdir persistent/solrcore
sudo chown -R 8983:8983 persistent/solrcore

sudo chown -R 100:100 persistent/iiifconfig/

docker-compose up -d

docker exec -ti esmero-php bash -c "chown -R www-data:www-data private"

ip address show

docker exec -ti esmero-php bash -c "composer install"
docker exec -ti esmero-php bash -c 'scripts/archipelago/setup.sh'
docker exec -ti esmero-php bash -c "cd web;../vendor/bin/drush -y si --verbose config_installer  config_installer_sync_configure_form.sync_directory=/var/www/html/config/sync/ --db-url=mysql://root:esmerodb@esmero-db/drupal8 --account-name=admin --account-pass=archipelago -r=/var/www/html/web --sites-subdir=default --notify=false install_configure_form.enable_update_status_module=NULL install_configure_form.enable_update_status_emails=NULL;drush cr;chown -R www-data:www-data sites;"
docker exec -ti esmero-php bash -c 'drush ucrt demo --password="demo"; drush urol metadata_pro "demo"'
docker exec -ti esmero-php bash -c 'drush ucrt jsonapi --password="jsonapi"; drush urol metadata_pro "jsonapi"'
docker exec -ti esmero-php bash -c 'scripts/archipelago/deploy.sh'
docker ps

Note the docker-compose did produce an warning message:

illuminate/filesystem suggests installing league/flysystem-rackspace (Required to use the Flysystem Rackspace driver (~1.0).)
Package zendframework/zend-stdlib is abandoned, you should avoid using it. Use laminas/laminas-stdlib instead.
Package zendframework/zend-escaper is abandoned, you should avoid using it. Use laminas/laminas-escaper instead.
Package zendframework/zend-feed is abandoned, you should avoid using it. Use laminas/laminas-feed instead.
Package zendframework/zend-diactoros is abandoned, you should avoid using it. Use laminas/laminas-diactoros instead.
Package container-interop/container-interop is abandoned, you should avoid using it. Use psr/container instead.
Generating autoload files
Carbon 1 is deprecated, see how to migrate to Carbon 2.
https://carbon.nesbot.com/docs/#api-carbon-2
    You can run './vendor/bin/upgrade-carbon' to get help in updating carbon and other frameworks and libraries that depend on it.
> DrupalProject\composer\ScriptHandler::createRequiredFiles
Create a sites/default/settings.php file with chmod 0666

I currently see running:

 docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                    NAMES
217ac267eb6f        nginx                              "nginx -g 'daemon of…"   22 hours ago        Up 22 hours         0.0.0.0:8001->80/tcp     esmero-web
40bf5a220f5e        esmero/php-7.3-fpm:8.x-1.0-beta2   "docker-php-entrypoi…"   22 hours ago        Up 22 hours         9000/tcp                 esmero-php
9cb8ef3b9819        solr:7.5.0                         "docker-entrypoint.s…"   22 hours ago        Up 22 hours         0.0.0.0:8983->8983/tcp   esmero-solr
a0b7a016c673        minio/minio:latest                 "/usr/bin/docker-ent…"   22 hours ago        Up 22 hours         0.0.0.0:9001->9000/tcp   esmero-minio
e48fc3288bd5        esmero/cantaloupe-s3:4.1.5         "sh -c 'java -Dcanta…"   22 hours ago        Up 22 hours         0.0.0.0:8183->8182/tcp   esmero-cantaloupe
b45b6a8fd16a        mysql:5.7                          "docker-entrypoint.s…"   22 hours ago        Up 22 hours         3306/tcp, 33060/tcp      esmero-db

[Diego Pino]

Hi, thanks for reaching out and great to see you got it running on Redhat, even if we had no docs for the docker part there. 

The demo deployment and config is 'rigged' to use the ports set in docker but thanks to the "devops" gods we managed to set, most of time, configs using the internal docker network ports and names so changes like this would not affect you directly. Means that 9001 port change on the public facing side of things should not make a difference (e.g see https://github.com/esmero/archipelago-deployment/blob/8.x-1.0-beta2/persistent/iiifconfig/cantaloupe.properties#L194) . But there could be other peculiarities in your setup/permissions, etc that could be affecting, (no firewalls right?)

Let's troubleshoot this:

- First check is: are you running archipelago locally (accessing the website via a http://localhost:8001) or via a real/public exposed domain/IP address? If the later, the solution to your problem is quite simple. Cantaloupe (IIIF manifests are accessed Externaly via Ajax/JS which implies the browser needs to be able to make the call. If your browser can not reach localhost:8183 (the setup cantaloupe URL on the demo) no images will be shown. Go to /admin/config/archipelago/iiif in your archipelago and change the URL for the public accessible server from localhost to whatever IP/public one you are using (don't remove the port) and save. Clear caches in drupal to be sure and try again. Not the issue? Let's move forward

- Did you setup min.io with the 'archipelago' bucket? If you log in minio at port 9001 do you see a bucket named archipealgo with folders starting with hashes? (like pieces of an md5 string?). If not you missed a step during deployment. If you missed that Archipelago will not have the default place where to store the files you upload and IIIF won't know where to find them to show them. Not an issue? You did not miss any steps and all looks there, there is at least one hash folder and it has inside, surprisingly, that JPG you uploaded (that would mean all went well). Let's move forward.

- Check your drupal logs first (/admin/reports/dblog) to see if there are any errors mentioned. Feel free to share with us. No errors? 

- Check if you have firewalls blocking ports 8183. 

- On that recently ingested Object, in its landing page, in the raw metadata display there will be an as:image json key with a lot of data there. That key should contain something like

"as:image": {
        "urn:uuid:c407e980-04b5-4add-9bc0-bd5ec7087e90": {
            "url": "s3:\/\/beb\/image-hamlibcom-23118869-c407e980-04b5-4add-9bc0-bd5ec7087e90.jp2",
            "name": "hamlibCom-23118869.jp2",
            "tags": [],
            "type": "Image",
            "dr:fid": 12,
            "dr:for": "images",
            "dr:uuid": "c407e980-04b5-4add-9bc0-bd5ec7087e90",
            "checksum": "beb8481ec9e6f70ad0dfa5c29cd99396",
            "sequence": 1,
            "crypHashFunc": "md5"
        }
    },

 

Most important is, the url should start with an S3. Check in your mini.io at port 9001 if there is inside the archipelago folder a has folder and a file name same as stated in your key.

- Lastly, if you passed all those sanity checks, and there seems nothing else is wrong, check your browser JS console and check what AJAX call (and the URL) failed, with either a code 500/401 and share that with us. Since all those calls are via ajax. But i kinda have the guts you missed the minio.io step or there is some type of other permission issue, given the fact that the thumbnail is not appearing and that is just an http call via html to iiif.

Can go into deeper debug mode with you in case nothing you find all to be working, even sharing screen.

let us know, happy to help further

PS: the deploy.sh script does not ingest demo objects/nodes (could, same logic, but we have not done that, in beta3 hopefully) but ingests demo metadata display entities, which are also content (see /admin/content) but contain twig templates used to transform JSON into HTML, JSON-LD, IIIF manifests, geojson in real time and used all over the place in archipelago, for formatters, api endpoints and other features.

best

[CSHL Library]

Thanks Diego for your troubleshooting steps. Everything checked out.

The tip off to the problem came from your comment about the thumbnail not appearing.

The browser was not finding the "default.jpg" image because it was looking at: "http://localhost:8183/iiif/2/...", whereas all of the rest of the URLs were referencing the actual server name/ip address. (Archipeligo is installed on a server, not on my local machine.)

I remembered in AWS install video from Zack of the Southeastern New York Library Resources Council, had a handful of pesky localhost:8183 to individually hand edit in the configuration files. So I executed the following, to update all of the configuration files:

cd archipelago-deployment

grep -rlZ localhost:8183 config/sync | xargs -0 sed -i 's/localhost:8183/MYSERVERNAME:8183/g'

sudo vi install/format_strawberryfield.iiif_settings.yml

I then needed to issue a "docker-compose down" and then redo the installation, starting with "docker-compose up -d".

Now back to reading: https://github.com/esmero/archipelago-documentation

Thanks

Tom

[Diego Pino]

Hi Tom,

Great! Just in case you need to change those paths in the future without syncing configurations or restarting, the global image server URLs (8183 ones) can be set at http://yourdomain.edu/admin/config/archipelago/iiif, most of the uses inside archipelago are managed from there centrally.  Look at that /admin/config/ under archipelago to see all the settings we expose. The config file replacement you did is also a good strategy of course, if you want to start with a clean installation from scratch when not running on localhost. Will make sure you document that too.

FYI: i’m working on an SSL/production deployment documentation  now, includes some different settings, SSL certificate generation, etc. Will publish on the group when ready.

Best!

Diego Pino Navarro

Assistant Director for Digital Strategy
Metropolitan New York Library Council
599 11th Av. New York, NY 10036

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/2f85d851-adb7-4100-9b4e-05bdb01afedd%40googlegroups.com.

[CSHL Library]

Hi Diego,

I noticed that cantaloupe, minio, solr, archipelago services, all seem to have some sort of admin/login functionality exposed.

Can you provide any guidance on how to harden the archipelago server, so that each of the services does not use the baked in username/password from the software or installation instructions.

There is a security update available for Drupal that is bundled with Archipelago, is it always safe to go ahead and install the security patches?  The update page implied that webform and drupal core needed to be updated manually. How can I do that from within docker?

Thanks.

Tom

[Nate Hill]

Hey Tom,

Diego has been multitasking and managing a ton of different things lately. I wish I could answer these questions, but I don’t know the answers myself. Hang in there, he will get with you. I’m glad to see you are spending so much time with Archipelago!

Nate

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/3c2599b5-1e11-4a4b-80a6-b13aa6e4bdf1%40googlegroups.com.

--

Nate Hill

Executive Director

Metropolitan New York Library Council

metro.org

917-375-0332

[Diego Pino]

Thanks Nate for covering for me! I really appreciate that

Hi Tom, good question! I think this can be even its own post, let's see how it goes

Archipelago/Security and Docker: Will itemize my answer so its easier to read

Archipelago deployment is not fine tuned for production. It can be quickly adapted to and i have been writing SSL/Production deployment guides in the last few days, plus some scripts and a new docker-compose yaml file. Good thing is nothing you have running already changes, just the way we connect things. And even so, i define myself a 'specialist in all styles' which really means i lack (most probably) some security deepness of pure DevOps Professionals, so there will be always better ways of doing things. That said

1- In our Docker Ensemble we tried to make most of the important interactions happen directly inside the docker network. Those connections are not exposed to the outside, basically calls in the form of http://esmero-web for example are never routed to your host machine. But, since archipelago-deployment was meant to develop, test, explore, we left many admin interfaces and non secure things around so you could access interfaces easily while debugging and learning. Which are those?

a.- Minio.io, at port 9001 (or in your case 9000). 

b.- Solr at por 8983

c.- Cantaloupe at 8183.

- Min.io access via host machine or via internet is not needed. Its handy but not needed. Archipelago accesse minio from esmero-web/esmero-php  and from esmero-cantaloupe, both using  esmero-minio name through the internal docker network

- Solr is accessed from esmero-web/esmero-php using esmero-solr name through the internal docker network

- Cantaloupe is indeed! accessed from internal network but also host, internet, the world, space and the larger universe because the fact that IIIF in our sense is meant to be open to the world (and there is AJAX use via the many Viewers like Mirador, IABookreader, etc) but you can either proxy it via ngnix (part of that guide i'm writing) and also disable the admin interface directly in the config (see down there to find the Container config files)

So, what can you do?

I) either close, disable every admin interface in each config file for each service, cantaloupe one is just a boolean, Solr not that easy to be honest. But, ports/sockets will still be open to the world!

II) make at least  esmero-solr, esmero-minio only available to the internal network (see esmero-db in the docker-compose file). Then proxy the admin interfaces via 443 or 80 and secure those on ngnix level.

III) use IPtables on your host machine (danger !) but its tricky since the way docker works is that it works at a higher level so IP table filters are mostly useless https://uilicious.com/blog/5-fatal-docker-gotchas-for-new-users/

IV) use a firewall that is outside your host machine, in AWS, GoogleCloud, Azure, etc there are many alternatives and we do filter most of these services via IP ranges so we still have access for our (mine) use but not for friendly hackers

V) Help us to explore other options and write documentation  (you can not blame me for trying!)

VI) use a Docker native firewall, do a google search, most are quite easy to setup and 

For I and II) , Where are all my Containers configs? (in bold config, almost all are there, you won't see certbot, that is my WIP for the ssl guide))

$ ~/archipelago-deployment/persistent$ ls 

certbot db iiifcache iiifconfig miniodata solrconfig solrcore

Except ngnix which is located here (you won't see that default-multisite.conf, that is my WIP for the ssl guide)

$ ~/archipelago-deployment/nginxconfigford8$ ls

default-multisite.conf default.conf xdebug.conf

Drupal updates:

yes. you can update and test, and report back. We do a quite extensive work, sometimes its even darker and heavier than coding testing the versions we provide via composer (see the composer.lock file). Reason is? Drupal devs and site builders are daredevils and like danger! We, on the repository side of things can not risk to make someone loose 1.000.000 objects because some contributed module maintainer had a bad week. So we test and test.

How to update? Always, always via composer. the Drupal UI is there just for old/sake/good times/reasons. composer is your friend and enemy. Don't update things via the website.

let's say you want to update strawberryfield/strawberryfield package. In your host machine type (and press enter after wards)

First (so you get used to)

docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer info strawberryfield/strawberryfield'

name     : strawberryfield/strawberryfield

descrip. : A strawberry field for Drupal 8

keywords : 

versions : * dev-8.x-1.0-beta2

type     : drupal-module

license  : GNU General Public License v2.0 or later (GPL-2.0+) (OSI approved) https://spdx.org/licenses/GPL-2.0+.html#licenseText

source   : [git] https://github.com/esmero/strawberryfield.git 1d3a3dd59c69465aaf8349463e5dd0a96937353c

dist     : [zip] https://api.github.com/repos/esmero/strawberryfield/zipball/1d3a3dd59c69465aaf8349463e5dd0a96937353c 1d3a3dd59c69465aaf8349463e5dd0a96937353c

path     : /var/www/html/web/modules/contrib/strawberryfield

names    : strawberryfield/strawberryfield

requires

drupal/search_api ~1.14

frictionlessdata/datapackage dev-master

ml/json-ld ^1.1

mtdowling/jmespath.php ^2.5

swaggest/json-schema ^0.12.25

So the composer info command will show you what you have . Also why the php -d non sense? Well, we try to limit the amount of memory we give esmero-php by default to 1024 Mbytes. Composer is a beast and because of package cross dependencies it builds a huge tree in memory and it fails with 1024 Mbytes. the php -d ... instruction just tells composer to use (while you are looking at it, so supervised!) whatever memory it can get hold of. So that way it works. There are other ways to avoid that command and just run composer something somepackage, but i like performance limitations to get things done.

Ok, to update that package under the same current version you run

docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer update strawberryfield/strawberryfield --dry-run' 

(see the --dry-run? it will do  nothing, it will just simulate what is needed).

If all goes well there, you can run

 

docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer update strawberryfield/strawberryfield'

Ok, but if you want to move to a specific version of Drupal? (you are on 8.8.2 right now).

docker exec -ti esmero-php bash -c "php -dmemory_limit=-1 /usr/bin/composer require 'drupal/core:8.8.5'"

That will bring 8.8.2 to 8.8.5. After that, clear caches (drush cr). Same goes for webform, etc. To be honest package dependencies is complex and that is why we provide that for you. IF you find issues, happy help, beta3, e.g will ship with every package we tested to its latest version.

Finally, for fun run this

docker exec -ti esmero-php bash -c 'php -dmemory_limit=-1 /usr/bin/composer outdated'

LONG LIST! Red/outdated does not mean, let's go for it! I always recommend updating similar group of things at the same time, testing with code and data, looking at logs, clearing caches (drush cr) and reading release notes. Newer just means, less tested. Also, always keep a copy of your old composer.lock and composer.json (version control)  Allows you to quickly go back to a working state by issuing a composer install in case of blue screens.

Hope this helps and someone can give me a hand bringing this into some documentation! Please follow up if you find issues and/or need more help or want to contributed things, knowledge or frustrations (success stories are also super welcome)

Take good care.

Diego

On Wednesday, April 22, 2020 at 8:44:01 PM UTC-4, Nate Hill wrote:

Hey Tom,

Diego has been multitasking and managing a ton of different things lately. I wish I could answer these questions, but I don’t know the answers myself. Hang in there, he will get with you. I’m glad to see you are spending so much time with Archipelago!

Nate

On Wed, Apr 22, 2020 at 1:49 PM CSHL Library <cshl.l...@gmail.com> wrote:

Hi Diego,

I noticed that cantaloupe, minio, solr, archipelago services, all seem to have some sort of admin/login functionality exposed.

Can you provide any guidance on how to harden the archipelago server, so that each of the services does not use the baked in username/password from the software or installation instructions.

There is a security update available for Drupal that is bundled with Archipelago, is it always safe to go ahead and install the security patches?  The update page implied that webform and drupal core needed to be updated manually. How can I do that from within docker?

Thanks.

Tom

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.

To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-commons+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/3c2599b5-1e11-4a4b-80a6-b13aa6e4bdf1%40googlegroups.com.