EntityStorageException Exception with s3 storage

[Kyle Huynh]

Hello, 

I successfully deployed Archipelago (https://github.com/esmero/archipelago-deployment) in Docker. Everything works great except I am unable to save Digital Object. 

To replicate: 

Content > Add Content > Digital Object or Digital Object Collection > Fill out required fields in Step 1, 2, 3, and 4 > Click Save Metadata > Step 6, save as Draft or Published > Click Save.  Then, the error below occurs: 

The website encountered an unexpected error. Please try again later.

Drupal\Core\Entity\EntityStorageException: The specified file 'temporary://filefImbMa' could not be moved to 's3://dostorage/5f59b1ea-457f-4f92-a20d-f18a10f27bb7/node_digital_object_5f59b1ea-457f-4f92-a20d-f18a10f27bb7.json'. in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 846 of core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

My S3 File settings is set as default  as installation guide above

:  
My /var/www/html/web/sites/default/settings.php: 

Also I already created a bucket: archipelago in MinIO as well. 

I am wondering if is there any  installation step which I may have missed. Thank you very much. 

Regards, 

Kyle. 

[dp...@metro.org]

Hi Kyle, thanks for the very detailed description.

That error as you correctly diagnosed is related related to S3FS not being able to connect to The minio container.

Since your Settings all look good and exactly as the shipped config should provide we need to start testing while the connection is not happening.

The S3FS module provides a "check" under actions. Here:

If you press Validate and all is good you should see that green mark. If not you will get a message I would ask you please to share here. You have not changed the minio password right?

Could you also share what platform (OS) you are running and which docker-compose YML file you chose for the installation?

If you went for OS X, where are in your filesystem are you deploying/git cloning the repo? One annoying thing about recent OS X versions is that it will try to Cloud Sync certain folders interfering in general with Bind Mounts and more annoying is that it will use a case insensitive filesystem, but with a recent Docker Desktop that should not be a larger issue.

 If for some reason you skipped one step and then you reapplied it afterwards, there are chances that the main setting that impedes proper connection is

$settings['s3fs.upload_as_private'] = TRUE;

If that one is not live (settings.php is cached) a cache clear of the full system may help. Should not be the case since you managed to get everything else running, but who knows?

Just for the sake of sanity I will do (yet) another deployment to make sure I can get a localhost running installed without any issues, so far I had not any problems but I want to be double sure. 

Please let me know how that "Validate" works out, and we can go from there.

Thanks again and welcome to the group!

Diego

[dp...@metro.org]

Hi Kyle, me again =)

So, could you also run this command for me?

docker logs esmero-minio 

Mine says:  You are running an older version of MinIO released 6 days ago 

All this because i just saw Minio published 18 hours ago a new minio/latest container and since Murphy's laws and OSS are best friends (and part of our family) I want to be sure we are both running the same minio. I'm right now testing the last update they pushed so will also report back on that if I find any bugs there.

More on that later.

Best

[dp...@metro.org]

Good news is I can't reproduce it, not even with the minio/latest from 7 hours ago which means it is not a global problem.

To test that I did a 

$ docker-compose down

$ docker-compose pull //which got me a new minio and an update DB container)

$ docker-compose up -d

Logged in as Admin,

I ingested manually and via AMI without errors and S3F3/Actions/Validate also passed.

Logged in as demo

Ingested manually (webform) a Video

No errors in the logs and images/videos can be seen

So, sadly bad news is I can't reproduce it which means there is something else going on there. Next step is to blame some OS/Docker local settings. 

Once you can share that data let me know and we can keep troubleshooting. Pretty sure we solve it and we learn something in the way!

Thanks!

[Kyle Huynh]

Hi Diego, 

Thank you for your quick reply, I have have ran the cmd: docker logs esmero-minio, and this is the output: 

archipelago-deployment % docker logs esmero-minio 

 You are running an older version of MinIO released 1 day ago 

 Update: Run `mc admin update` 

Endpoint: http://172.27.0.3:9000  http://172.28.0.3:9000  http://127.0.0.1:9000 

Browser Access:

   http://172.27.0.3:9000  http://172.28.0.3:9000  http://127.0.0.1:9000

Object API (Amazon S3 compatible):

   Go:         https://docs.min.io/docs/golang-client-quickstart-guide

   Java:       https://docs.min.io/docs/java-client-quickstart-guide

   Python:     https://docs.min.io/docs/python-client-quickstart-guide

   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide

   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide

IAM initialization complete

I'll run your most recent post shortly. Thanks. 

Kyle. 

[Kyle Huynh]

... I tried the validation on the S3 File System as you suggested. I found this error, maybe I need to change the permission of private directory (set as "file_private_path" ) in the settings.php ?  Thanks. 

On Tuesday, May 18, 2021 at 3:40:09 PM UTC-4 dp...@metro.org wrote:

[Giancarlo Birello]

Just an idea more, does S3 settings server url need a /minio too? Sorry friends if this is not useful, almost sleeping here

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/21762529-b8e6-4550-a20d-c439065ef528n%40googlegroups.com.

[dp...@metro.org]

H Kyle, that error matches the one you see when the $settings['s3fs.upload_as_private'] is missing or not = TRUE. But your settings.php has that.. mmm, but also that code means data transfer interrupted so may be that 1 day old bad bad minio release.

(what a timing!)

Let's do the following steps:

1. Please confirm you have that setting and there is no Docker File sharing discrepancy/error because of your OS or Docker Desktop

docker exec -ti esmero-php bash -c "tail -n 100 web/sites/default/settings.php|grep s3fs.upload_as_private"

You should see

$settings['s3fs.upload_as_private'] = TRUE.

2.- Stop all your docker-compose ensemble and update to latest minio (because the one from yesterday was buggy)

docker-compose down

(wait until it finishes)

docker-compose pull

Should get you a new minio (minio will download and finish)

docker-compose up -d

Check the S3FS action/validate again.

3.- If nothing works still, please share with me

- Your OS and your version/platform (if you are using a Mac with an M1 chip please let me know, those 'things' require the latest Docker Desktop)

- Your Docker Desktop and which File Sharing Mechanism are you using (Under Preferences/General)

- Where in your Drive is archipelago-deployment located (do a pwd)

Crossing fingers 2.- fixes the problem. If nothing else works also happy to have a Zoom call and do a live troubleshooting, I'm quite curious now!

Hugs

Diego

[Kyle Huynh]

.... I tried part 2 as out below: 

archipelago-deployment % docker exec -ti esmero-php bash -c "tail -n 100 web/sites/default/settings.php|grep s3fs.upload_as_private"

$settings['s3fs.upload_as_private'] = TRUE;

archipelago-deployment % docker-compose down

Stopping esmero-web        ... done

Stopping esmero-cantaloupe ... done

Stopping esmero-solr       ... done

Stopping esmero-minio      ... done

Stopping esmero-db         ... done

Stopping esmero-php        ... done

Stopping esmero-nlp        ... done

Removing esmero-web        ... done

Removing esmero-cantaloupe ... done

Removing esmero-solr       ... done

Removing esmero-minio      ... done

Removing esmero-db         ... done

Removing esmero-php        ... done

Removing esmero-nlp        ... done

Removing network archipelago-deployment_host-net

Removing network archipelago-deployment_esmero-net

archipelago-deployment % docker-compose pull

Pulling php   ... done

Pulling solr  ... done

Pulling web   ... done

Pulling db    ... done

Pulling nlp   ... done

Pulling iiif  ... done

Pulling minio ... done

archipelago-deployment % docker-compose up -d

Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "archipelago-deployment_host-net" with driver "bridge"

Creating network "archipelago-deployment_esmero-net" with driver "bridge"

Creating esmero-php        ... done

Creating esmero-nlp        ... done

Creating esmero-cantaloupe ... done

Creating esmero-db         ... done

Creating esmero-solr       ... done

Creating esmero-minio      ... done

Creating esmero-web        ... done

The download of MinIO not happened because I already run it earlier  and I saw it ran the downloaded the latest version. 

I also ran the validate again, but the same error:

"Unable to validate your s3fs configuration settings. Please configure S3 File System from the admin/config/media/s3fs page or settings.php and try again.

An unexpected error occurred. Error executing "PutObject" on "http://localhost:9000/archipelago/s3fs-tests-results/write-test-180521-1657.txt"; AWS HTTP error: cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)
Could not upload file as publicly accessible. If the bucket security policy is set to BlockPublicAcl ensure that upload_as_private is enabled in your settings.php $settings['s3fs.upload_as_private'] = TRUE;

Error message: Error executing "PutObject" on "http://localhost:9000/archipelago/s3fs-tests-results/public-write-test-180521-1657.txt"; AWS HTTP error: cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)"

As Giancarlo suggested above, maybe should be "/minio" in "http://localhost:9000/archipelago/s3fs-tests-results/public-write-test-180521-1657.txt",  is the "/minio" needed to be in the Hostname in Settings Tab ? 

Also, 

- Your OS and your version/platform (if you are using a Mac with an M1 chip please let me know, those 'things' require the latest Docker Desktop) ==> Mac book pro (2020), OS: Mac OSX Big Sur, version 11.3.1

- Your Docker Desktop and which File Sharing Mechanism are you using (Under Preferences/General) : 

- Where in your Drive is archipelago-deployment located (do a pwd): ~/development/archipelago-deployment

[Kyle Huynh]

Oops, Docker General is which you asked for : 

[dp...@metro.org]

Hi again,

Ok, last attempt before we have do to surgery!

- Since you are running a modern OS X with a modern Mac and you have deployed under the $HOME folder big chances are Cloud sync/OSX Filesystem may be interfering.

- To simulate your Deployment I copied from one of my OS X partitions a fresh deployment (from one hour ago) into my own $HOME folder and funny enough, I had no errors (but I kinda knew why it would not fail the bucke had been already created)

So I started fresh there, I deleted the Bucket. Stopped docker-compose, started it again and created the buck all over again... but I even if the bucket got created a I saw a lack of response from the UI and a spinning wheel in the UI (which I figure it was a similar error, client (UI) not talking correctly to the Minio.

Ok, so docker-compose down again. I had Use gRPC Fuse dis

abled, I enabled it and started again. S3FS validated and I could see everything/ingest.

But: gRPC FUSE is deceiving. Depending on your Desktop version and your OS X combination it can be "good" or it can be "bad". 

So, let's normalize:

1. -Same  Docker desktop?

I have this one. You on a newer or older? My experience with Docker Desktop has been that every time the change the code of gRPC Fuze and depending on the OS version (I run Catalina 10.15.17 on a Mac mini) Fuze either does harm or the opposite.

This is what I have now.

- So, if you on an older please update and restart and test all over again. You may even recreate the archipelago bucket if needed

- If you on the same one, disabled Use gRPC. Apply and restart. Log into Localhost:9000 , remove the archipelago bucket. Create it again. Add a file via the UI there directly (a small one) see if it saves correclty. Try with S3FS ACTION/VALIDATE again.

- Additional: just to discard other things. You have an external drive just to test? you can cp -rpv archipelago-deployment into that one and try  a docker-compose up there.

We have people running Big Sur without problems so far but there may be other combinations in place

If nothing of this works, let's do a live session. I can anytime tomorrow before 12 PM EST time or after 16EST.

So sorry I can't replicate the exact behavior but we will figure it out!

Best

Diego

[Diego Pino]

Also, when you said Apple MacBook Pro 2020, you mean the Intel i7 one of the November M1 (ARM) Chip?

If M1 we may be getting into Architecture issue / Docker Desktop / Rosetta problems and you may need to switch Docker Desktop to Edge and I will need to build multi Arch containers or AMD/INTEL/ARM and PPC (since I’m there). Minio is already multi Architecture but our custom PHP/Cantaloupe containers are not.

Another way of testing if MINIO is the issue (filesystem/bind mount) or even the PHP container is to manually upload a file to the archipelago bucket via the minio UI.

Anyways, happy to do that live session and figure it out. I even deployed a Linux version and all worked out, so really really curious.

Thanks for sharing this

Diego

Diego Pino Navarro
Assistant Director for Digital Strategy

Archipelago architect
Metropolitan New York Library Council
PO Box 2084
New York, NY 10108 

On May 18, 2021, at 5:51 PM, dp...@metro.org <dp...@metro.org> wrote:

Hi again,

Ok, last attempt before we have do to surgery!

- Since you are running a modern OS X with a modern Mac and you have deployed under the $HOME folder big chances are Cloud sync/OSX Filesystem may be interfering.

- To simulate your Deployment I copied from one of my OS X partitions a fresh deployment (from one hour ago) into my own $HOME folder and funny enough, I had no errors (but I kinda knew why it would not fail the bucke had been already created)

So I started fresh there, I deleted the Bucket. Stopped docker-compose, started it again and created the buck all over again... but I even if the bucket got created a I saw a lack of response from the UI and a spinning wheel in the UI (which I figure it was a similar error, client (UI) not talking correctly to the Minio.

Ok, so docker-compose down again. I had Use gRPC Fuse dis

abled, I enabled it and started again. S3FS validated and I could see everything/ingest.

But: gRPC FUSE is deceiving. Depending on your Desktop version and your OS X combination it can be "good" or it can be "bad". 

So, let's normalize:

1. -Same  Docker desktop?

I have this one. You on a newer or older? My experience with Docker Desktop has been that every time the change the code of gRPC Fuze and depending on the OS version (I run Catalina 10.15.17 on a Mac mini) Fuze either does harm or the opposite.

<Screen Shot 2021-05-18 at 5.43.21 PM.png>

This is what I have now.

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/1a1d0d2e-9157-4a17-9eae-12ce190936e4n%40googlegroups.com.
<Screen Shot 2021-05-18 at 5.43.21 PM.png><Screen Shot 2021-05-18 at 5.34.24 PM.png>

[Kyle Huynh]

Hi Diego, 

I found that my Docker Desktop's version was 3.3.0, so I upgraded mine to the latest version which is same as yours 3.3.3. and then the error was still there. 

But I realize the archipelago-deployment repo in my local was out of date as well, so I did git pull and see that there are new commits on today. Then, I decided to do a complete remove and reinstall the whole stack again, and IT WORKS ! :-). 

One thing which I found which different is the host name for the S3 system: http://esmero-minio:9000 by default. I think I might have changed to http://localhost:9000 to match with the URL when I setup the bucket. So this may be my fault at the beginning  OR the today commits may fix the issue. 

Anyways, everything works now. I am able to save a Digital Object. Thank you so much for troubleshooting with me today. :-)

Regards, 

Kyle. 

[dp...@metro.org]

Ups, I sent this without copy to the group last night so here it is for the sake of transparency =)

Hi! Wonderful news. I was already expecting to stay all night trying to figure out what was wrong :) (Can’t help it we really want people to be able to use archipelago!) Thanks again for sharing this with the group, following all instructions and reporting back your success! 

Please let us know if you have any other issues (any time) and please feel free to join the slack channel for some office hour “fun”

Hugs!

Diego

[dp...@metro.org]

H Kyle, a tiny note here:

the hostname is just a facility in case you need to publicly access the files from the internet but won't affect you at all. Since we really discourage public links for the files and we assume if you go public with an archipelago you will be also running a proxy for minio or even totally hiding it, what you put in the hostname does not affect you at all.

In other words, S3FS will take that hostname and append it to the File URL when using the Content-> Media list to allow (but won't work) to click on the files.

Archipelago allows the access of the files to happen differently by exposing internally an access controlled endpoint that "mimics" a IIIF endpoint

In this object here https://play.archipelago.nyc/do/86a37eb3-a539-4fc7-8353-802aced81a2f

you can see how the Twig Template (Object Description) exposes the internal S3 files as direct link downloads (scroll down).

I will also update the OS X docs to make sure we mention the exact proven versions of Docker Desktop for Big Sur and add some trouble shooting notes to the Archipelago Documentation for this use case.

Thanks again and have a lovely day

Diego

On Tuesday, May 18, 2021 at 8:34:11 PM UTC-4 kyle.h...@gmail.com wrote: