Some updates from the end of summer Lab for RC3 (and 1.0.0)

[dp...@metro.org]

Good afternoon,

It has been a while since I posted in our group. Because of time, summer, weird 2021 and well, time again, we have been focusing a bit too much on Slack interaction, but that does not mean we have forgotten public facing communication and most important we have not been lazy!

I will share with you all a few exciting advancements and will probably (again time constraints) add more each week so you can talk/discuss/ask or just contemplate where we are heading to on our next (and final release before 1.0.0)

Key Name Provider Overview form:

Yes, bad name. But an important development. As many of you know, we have a peculiar way to expose nested data/metadata coming from your Strawberry field JSON to the Drupal environment and we this approach JSON Key Name Providers. These little Plugins have been around since the first iteration of Archipelago (almost 3 years!) and are basically dynamic queries against your JSON that expose natively to Drupal pieces of flat data for Solr/fields/facets/etc consumption. You can add new ones, edit their extraction strategies, tell Drupal the values need to be parsed as Dates (includes super smart ETDF extraction and ranges, even timestamps), etc and you have basically full control on what you expose out and to what those values are casted (e.g a number to a NODE). Still, the fact that your Single Field produces multiple properties that lead to multiple Solr Fields and then each field to many many Facets may confuse anyone. And Drupal has this strange (but we get it) way of not exposing how one thing is connected to another via UI. Basically Admin UI is atomic and only shows only tiny piece of the puzzle. So we did something to aid in this, an overview display that shows your data from start to finish and allows you to access each Drupal provided UI interface in a context aware fashion. Better with a screen recording (VIDEO/MP4, small one I promise, look at the end of this post)

This is done using D3.js. It shows a tree starting from a Strawberry Field, then all the Key Name Providers with all the Properties that derive from each one followed by the Solr Field that are feeding from these properties/values and then Facets that are implemented from those Solr Fields. You can select any ADO (Archipelago Digital Object) from your repo and simulate the generic tree with real data, which instead of just showing the properties will also show live what data is being passed from the ADO into those and show extracted/processed values in a tooltip. The UI needs still some work (we are on it) and will have some legends, more color and some better text formatting. We also plan on doing the same for Your Twig templates! so you can see how each one issued, where and how. Cool!

CSV Export

Part of the AMI module. We developed a new Action that uses any Views Result (e.g the current Search And Replace one but can also be your normal/general search if you want o) to select ADOs you want to export (100s or 1000s if you want). Then gives you a few choices about how you want to manage the data (e.g If exported files will be ready to be reused values in an AMI Set for editing/updating ADOs or to ingest all in another repository) and also allows you to generate a new AMI set out of the export. Basically a Tabular data round trip. It does also some normalization of the CSV and works pretty well! This action comes to complement the Search/replace metadata as text, Search/replace using JSONPatch (advanced) and Webform/ Form Element based Search and replace (covers right now 80% of the elements and may need some sleepless nights to have it all right). AMI has also now an LoD reconciliation interface, where pure labels can be passed through a set of vocabularies and LoD endpoints and will give you back a CSV (yes!) but also ready to be used value/url keys that can be used in your AMI Ingest Templates to fill up any LoD needs you may have. Of course Labels won't match always 1:1 a single Term in wikidata/LoC/snac/Mesh/Getty/Europeana, etc, so an experimental feature was added that allows you to work on/refine via the UI the automatic matching done by AMI. All this can be found at the AMI set level now (extra tabs)

Again a small video at the end of this post

All these right now are Open pull requests and will pass some strict testing/UI feedback and de-bugging in the next days.

Format Strawberry field:

3D Models take now UV Map Texture and will too also read MTL for OBJ. The UI needs some extra work but some of the decisions made here for that Formatter will apply to every other one that uses multiple files as source for a viewer (e.g Video/Audio with transcripts), basically you will be able to select not only how many media elements each ADO provides but also where it was uploaded too, allowing a much better/smarter way of generating complex Objects where you have many different media types but only a few go together (matching sets of video + transcripts) or some are meant for other display needs (e.g large image used as preview of a 3D versus UV Map for the actual 3D). 

Also, Mirador 3 was patched (big hack) to properly manage Video/Audio updates on multiple Desktops with Audio/Video/Images mixed Manifests. Mirador is becoming quickly a main viewer for most of the type of assets/ADOs we provide, from Collection listings, Video, Audio, Documents, etc. And because of this, some deep work will be done to update all Twig templates that drive IIIF Manifests V3 to accommodate new capabilities. Giancarlo also has an open pull for ALTO OCR endpoints which will allow in a near future to have Mirador Text / OCR highlights and also of course Annotations not longer only on Open Seadragon.

Ok, long post (again Videos are attached, not worthy of Youtube yet!)

Thanks for reading, more soon (also sorry for the typos)

Best!

Diego Pino