missing something with the ami configuration, maybe?

[George Biggar]

Hi Folks,

We've been unsuccessful using AMI so far. Not sure what else to try.

Errors I've seen:

Error: Call to undefined method Drupal\ami\Plugin\ImporterAdapter\SpreadsheetImporter::getConfigurationForm() in Drupal\ami\Form\ImporterAdapterForm->form() (line 141 of /var/www/html/web/modules/contrib/ami/src/Form/ImporterAdapterForm.php)

TypeError: Return value of Drupal\ami\Entity\ImporterAdapter::getPluginConfiguration() must be of the type array, string returned in Drupal\ami\Entity\ImporterAdapter->getPluginConfiguration() (line 134 of /var/www/html/web/modules/contrib/ami/src/Entity/ImporterAdapter.php)

Thanks in advance, for your help and advice. 

Best,

George

[Diego Pino]

Hi George. Is this your local instance or play? Is the first its this a recent install? 

We have been doing some user testing/documenting in the last few weeks and are right now making sure we have all the exceptions (mostly missing configurations/setups on each instance)  checked and fixed. I will look at that error (line 141) and see if I can trigger it on a clean archipelago and if its an error and not just older code I will fix it.

Also happy to jump by the end of the week into a short call and give you a walkthrough because Allison has been ingesting a lot lately with it.

Stay tuned, working on it, will come back with answers by tomorrow morning

Diego Pino Navarro
Assistant Director for Digital Strategy

Archipelago architect
Metropolitan New York Library Council
PO Box 2084
New York, NY 10108 

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/677f659a-0316-4efb-995b-8796df86f90cn%40googlegroups.com.

[Biggar, George]

Hi Diego,

 

Thanks and no rush. This is a local installation. But, I followed the documentation and Zach’s video and configured for SSL. That’s working fine. Still behind the external campus firewall. Ubuntu 20.04. I think most everything else is working well.

 

Please, seriously, no rush.

 

Thanks and best,

Geo

[dp...@metro.org]

Dear George, took me a while but I know now what is happening. OK, this is lack of documentation on our part and code not yet implemented (more like not tested...) for a very specific use case. Will explain

still, since AMI is not the release run yet (means it has things working quite well, others on the test, others we know not, but coding for them and options exposed that will do not do anything...but believe me it actually works and you can ingest mass amount of stuff) 

I recommend that you first subscribe to the GitHub repo (watch) because it will evolve faster during the next weeks to cover the missing parts and also to get into a versioning number roadmap.

If you are running a recent (last few weeks) 1.0.0-RC1 deployment AMI should be using this in your composer file "archipelago/ami": "dev-main", which is this GitHub branch "main" at

https://github.com/esmero/ami

First the error. Wrong spot for now. Do not use (yet) this http://localhost:8001/admin/structure/importeradapter. That interface is not meant (yet) for actual ingest but to configure "canned" ingest strategies for users with less credentials. The idea is that eventually you/someone will do all the mapping/configuring that AMI needs to be able to read from a CSV and generate the data you want. And you will not want other users to have to go through that so that section will allow you to create a single config that will be exposed to those users as a single "Ingest your Super Set" link/button and will do a LOT of checking on the CSV, removing the flexibility of this to enforce needs/ways. 

But since we changed a lot in the actual ingest step, its not "in touch now" with the rest and needs cleanup. Which I will do today!

Also, let's do a screen share and we can record a video since batch ingest is actually nice and fast. In the meantime some info on what works.

Alert: long post ahead, read only if you want to. Every of my post can be replaced by a Video or ongoing better docs.

Ok, to our (long) business (what do to)

The workflow that is tested as now (without setting special things up)

1.-  You go to Admin / Content and you will see  a button with a + sign (blue as admin) that says "start an AMI set"

2.- That will give you a multi step form. Select "Spreadsheet Importer", press next

3.- Select "Create New Ados" from the operation (the other modes are trickier and deserve a Video or a full Musical from our part + testing scenarios so you do not need to guess)

4.- Upload a CSV (question is how should that CSV look like? Good question George. Allison will share/publish an example CSV with a few rows that fetches assets remotely and also uses your already present demo assets and with some basic metadata later during the week, she is very busy right now testing our code so we can fix/improve and do better but CSV will be shared!), but the gist is there is not a single CSV that works. It depends on how each ROW is going to be mapped to your expectations (expectations in the form of JSON and files). There is a demo twig template in the RC1 deployment for that. Super simplistic that assumes that every column can be either a simple String OR an actual JSON snippet (its the lowest processing of a twig template) but meant to be verbose about what is happening.  Basics are:

4.1 Columns can be mapped to data needs. Some can point to files (if many separated by ;), and you can have many column, we recommend one per type of tile "images", "documents", "models", "videos", "audios". These values an be a path to an existing file accessible to your docker container. e.g your d8content folder (which internal will be seen as /var/www/html/d8content/, so if you copy a file named cat_001.jp2 there the cell in the spreadsheet under the "images" column Neds to be /var/www/html/d8content/cat_001.jp2, it can be also a full url (e.g CAT URL here.. paste plain in the spreadsheet) and Archipelago will get it for you, or if can be an S3 path too. s3://georges_folder/cats.jp2 (you can upload that one to your local minio at localhost:9000 inside a cats folder (hint, if in S3 archipelago will respect the place and will not try to move it somewhere else, the can be good but also warning, bad. We may want to make that an option in the near future)

4.2 Some columns are needed 

- node_uuid (can be empty, this is for migrations in case you want to keep the UUID, if not archipelago will give you one
- type (Book, etc, anything you normally would set via a webform)
- label (The title of the Object)

4.3 Some are optional but recommended. 
- ismemberof
- part_of

- etc (etc because you can connect objects using any Object to Object relationship that matches your needs/predicate)

This columns can hold 3 types of values: 

- empty (like nothing not the word empty) to make an object member/connected to nothing)

- an integer, if the object will be connected to another row in the same Spreadsheet (if you ingest a collection in the second row - first are the headers - then a third row can have the number "2" in its ismemberof. On ingest archipelago will make that row 3 a member of the collection generated on row 2.

- an UUID. If UUID is used it will connect to an already ingested Object

- 4.4 metadata. Every column header will become a JSON Key and each cell value a JSON value for that key. You can use directly JSON snippets or do crazy logic (like putting a lot of values separated by semi colons and then use the twig template *you will select in another step* to split the values, capitalize them, clean them, many cool stuff and generate the correct JSON structure.

5.- After that a lot more happens and you get to select mappings, approaches etc. This will end being a too long post so I think we could have a tiny screen sharing session and record a video and share with everyone. Still, please give it a try/test options, twig template mapping and direct are both working, I would to see what happens when someone is exposed to the UI without docs (sorry for the experimentation) 

6.- If you manage to get through all the steps you will see a ZIP upload (no need to upload just press , legacy thing we will probably implement soon to allow also have a ZIP source for your assets as we do in our predecessor module). At the end you will generate an AMI set (which is a bundle of your CSV enriched and prepared, the settings you used, etc). That set now can be ingested. uFF!

Long, Sorry

Allison and I will get back with some dates for a full tut and will have the docs by Monday (pinky promise)

Thanks!

D 

[Biggar, George]

Hey Diego,

Thanks so much for your fast and thorough reply. The good news is we made progress. The problem was that we were uploading a zip file of objects. Once the files were moved and the path was set correctly in the csv file, the ami set succeeded. We’ll continue with more testing. And a huge wow for all the great details. Very much appreciated.

 

Thanks too to both you and Allison.

 

Best,

Geo

--
You received this message because you are subscribed to a topic in the Google Groups "archipelago commons" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/archipelago-commons/DV3tu_b-_FA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/06015dad-a6c4-4538-8e83-6f06595a4a0an%40googlegroups.com.

[dp...@metro.org]

Thanks George. I will be working during the weekend on some improvements. In my immediate list is a few contextual links to make Batch processing more visible, exposing Logs (success, issues, etc) directly on the AMI set and also (of course) the ZIP handler. Have a great weekend and thanks for using, testing and discussing AMI and/or any part of Archipelago with the community!

Hugs

[Allison Lund]

Hello George, everyone,

Hope you had a nice weekend & that your week started off on a good note!

The first round of AMI documentation is now available at:
https://github.com/alliomeria/archipelago-documentation/blob/1.0.0-RC1-AMI-docs/docs/ami.md

The following guidelines, examples, and instructions are included:
- Brief general AMI introduction
- Google Sheets API Configuration
- Spreadsheet Formatting Overview (some of this language will feel quite familiar)
- Example Spreadsheet/CSV
- Example JSON Ingest template
- Instructions for Ingesting New Digital Objects using Spreadsheets or Google Sheets
- Instructions for Ingesting New Digital Objects and Collections using Spreadsheets or Google Sheets

George (and anyone else who may be interested), please take a look and let us know if you feel these docs read well, and if there are any particular features/functionality that would be helpful to better cover. Your feedback and time would greatly appreciated, but of course no pressure.

As Diego has already commented, AMI is an evolving tool, with enhancements already in the works for future releases. Please stay tuned for these updates, consider following the latest (or opening new!) issues at https://github.com/esmero/ami, and don’t hesitate to reach out with more questions, comments, or suggestions.

If you would like to keep your local AMI up to date with the latest code, you can always run the following command:
docker exec -ti esmero-php bash -c "composer update archipelago/ami"

The main Archipelago documentation, @esmero/archipelago-documentation, will be updated pending initial review completion for these AMI docs (thank you in advance Diego & Pedro!). Moving forward, we will do our best to keep documentation up to date with the latest code releases as much as possible.

Thank you very much for your time!

Take care,
Allison

[Brenden McCarthy]

Thank you so much Diego and Allison!

To follow up with Diego about being exposed to the ui without docs, we were able to successfully use the AMI with the above post and the UI. Used the AMI to ingest a collection of our RPI players Friday, screenshot below (I know Diego was curious!):

Thanks Allison, for this great AMI documentation (and especially the example spreadsheet), I know we'll be able to do a lot more now. I have just finished reading through all the docs and I think they read well and make perfect sense and are super easy to follow (I can follow up once I do some more AMI work alongside them if you're interested. So far mostly been doing a basic scans/csv route-- going to also try with google sheets once I configure the api tomorrow).

Thanks again for all the help and support,

Brenden

[dp...@metro.org]

Oh oh oh. Brenden and Geo. This makes us so happy! (more + more ohs)

Thanks so much for sharing

best

[Allison Lund]

Wow RPI Libraries team! Thank you very much for sharing! :)

Brenden, sounds great about additional follow-up as you continue with your AMI work (as long as that is not a burden to you in any way). Please let us know how your google sheets api configuration goes, and/or if you have any additional questions. Thanks again for your time and feedback.

Everyone, the main Archipelago documentation has now been updated to include the AMI docs, accessible here:
https://github.com/esmero/archipelago-documentation

Take good care all, stay safe and warm!