New user: having trouble running locally

[Alexandra Dolan-Mescal]

Hello all!

Trying to run locally on a new Macbook Air (M1 chip) - read through the docs, and edited docker-compose-osx.yml with the following:

platform: linux/amd64

Not doing the trick for me - Still returning this error: ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries

I'm only semi-literate in command line, so any guidance would be very helpful!

Thanks,

Alexandra

[dp...@metro.org]

Hi Alexandra, 

Those shiny M1 chips gosh!

So, short (first) story is the rosetta compatibility mode should allow you to run under "emulation" but it requires Rosetta and the latest Docker Desktop using the "Experimental Features" should allow the platform: linux/amd64 but still mysql 8 won't work correctly or IF, super slow.

So the second (short but not as short) story is Max (from Western) https://github.com/esmero/archipelago-deployment/issues/110  got it working fine on the new ARM M1 Chip by swapping MYSQL 8 for mariadb and building the containers locally and I will do officially now: rebuild the other containers (PHP/Cantaloupe/etc) and add new tags using something named buildx (multi platform) so you have not need to go through that pain.

Give me a few hours (probably my tomorrow AM but may be sooner?) and I will share a full replacement docker-compose-m1.yml (drop in solution) file that you can use out of the box without the hassle. (crossing fingers)

We have no M1s here (too fancy!) but I can try with another AWS Arm based server (m6 series) to make sure it works,

Stay tuned!

Best

Diego

[Alexandra Dolan-Mescal]

Thanks so much Diego! I did install Rosetta and mariadb before reaching out...but couldn't figure out what to do from there. I'll be on the lookout for the new M1 YML file!

Alexandra Dolan-Mescal

some art

213.309.1213

--
You received this message because you are subscribed to a topic in the Google Groups "archipelago commons" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/archipelago-commons/pBcQDcn6hN8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/78da1706-f8ee-476f-a9cf-fd33d693ff92n%40googlegroups.com.

[dp...@metro.org]

Perfect.

PHP and cantaloupe are built and pushed *succes* and one (NLP) is taking a bit longer but will be ready soonish. Once I have tested (to certain extend) I will share the new YAML and will ask you please to report back if you see/find any issues. More tomorrow, Thanks!

[dp...@metro.org]

Hi Alexandra,

Almost success. Rebuilding Java based Docker Container ones for M1 one (Cantaloupe) because whatever I did failed and all the base OpenJDK containers are not multi arc. Probably 2-3 hours more. But we are moving forward and hopefully this will solve it for good for future  needs.

Thanks for the patience

best

[dp...@metro.org]

Hi,

So sorry for the delay! 2-3 hours was deceiving. Happened that M1 chip support (ARM64/V8) was a bit more than rebuilding containers (and I learned a few new things). Will start by sharing the new docker-compose/solution first and then (for community to enjoy/suffer/ignore) what was done and what can be done better.

Solution: New docker-compose.yml for your 1.0.0-RC2 deployment can be found here: https://github.com/esmero/archipelago-deployment/blob/1.0.0-RC3/docker-compose-arm64.yml

You can either copy/paste the content of that file into a docker-compose.yml file or download and rename it as such.

Once you have it, run

$ docker-compose pull 

and you should see something like this after a while (new Images, a bit large for my taste but once you can confirm this works I will work hard to make them smaller)

Pulling php   ... done

Pulling solr  ... done

Pulling web   ... done

Pulling db    ... done

Pulling iiif  ... done

Pulling minio ... done

Since you are on OS X the rest of the instructions are (should?) be the same and all will(may?) run fine and you will be able to get an empty RC2 repo that with DEMO objects ready to be ingested using the AMI (instructions on the readme) after 10-15 minutes.

I disabled on that DOCKER COMPOSE YAML file the Natural Language Processing 64 server since its 700 Mbytes in size and its only used right now on Active RC3 development branches (Strawberry Runners V0.2.0, etc). By the way demo here https://studio.archipelago.nyc/search_websites?search_api_fulltext=

Tip: once you do a docker-compose up -d wait a few seconds and do a docker ps (do it a few times). No container should be "restarting" and the uptime of all should increment every time you run it. If NOT  I made a mistake and this was not such a good day (and I can fix/debug)

Now the tech-ie part nobody is waiting for.:

- The new FPM PHP container was easy. It is now multi architecture and can be used on x86, AMD, ARM64/v8 and even on a raspberry pi. 

- But the IIIF server and NLP ones? GOSH!

- Cantaloupe: Happens that our base containers (slim, small Alpine Linux) for our custom builds can not run modern Java on ARM64 because it simply does not compile (different libc library). So JDK support only for Java 8. (and we need 13)

So I made a large new Container based on Debian and recompile every little bit (many times because I'm flawed/fail and not optimal) and do some source code ARM specific things. So, Graphics Magic was recompiled for this, JP2 support on Debian and even if the container is 90% different than the x86 one is does the same. For Linux users there is a change in permissions which is explained on the YML and I will document when RC3 comes on the Install instructions. Nice thing is the maintainer of Cantaloupe updated the release with a fix related to an issue we opened exactly during the build so that went in (now its really 4.1.10 even if it says 4.1.9RC2-arm64)

For NLP things were even stranger. Some pip/python libraries where Hardcoded for x86 (compile flags) so I cloned repos, edited setup.py and changed the deployment strategy. This only applies to RC3/edge users of course and I used the time to add more languages for polyglot and spaCy (Portugues, French, German, we had Spanish and Italian already =). This container is large. 700Mb and I need to reduce its size soon.

In conclusion, both languages made for "portability" JAVA and Python. were of course not that portable at least for our needs.

I Tested this all on an AWS EC2 m6g machine with is an ARM64 Gravitron Processor based Instance with only 4 Gbytes of RAM (to stretch things a bit) and all works perfectly (will also add to the readme.md instructions for deployment on that docker-compose has no release and you have to build it... )

Alexandra, please let me know if this works for you. Max if you around please test if you can, this new containers also apply for you specially the Cantaloupe one.  Happy to iterate over this a few times if there are issues / things to document that I missed. 1.0.0-RC3 will have official M1/ARM64 support with docs.

Hugs and I hope you enjoy your local deployment

Diego Pino