Archipelago 8.x-1.0-beta2 open for everyone that craves for a sweeter, low carb repository system

[Diego Pino]

Hola, hello, hallo, Ciao people

As i silently suggested 2 weeks ago, Beta2 is out. In this last 14 days we made extra improvements, some cleanup, config updates and i'm proud to announce that its about time to get this officially out and shout it lout so testing, breaking and having fun can begin,

I really feels like version 2 and not just a beta, but we are pretty much perfectionists and what for others could be a full version, for us still feels like the start of a long road. But somehow that feels like a safe, comfortable spot to be too (and honest). Like looking at the horizon while we figure out how to swim that far.

The number of features we could squeeze into the code and configs is non the less impressive and we really, really worked hard to bring you some fun, some improved known functionality and some new things that will make metadata and DevOps people happy alike.

To the facts my friends:

Deployment strategy is still the same can can be found at https://github.com/esmero/archipelago-deployment/tree/8.x-1.0-beta2

Deployment Readme.md is updated to reflect the version changes. Still 5 Steps, but more things happen while you wait.

What is new in deployment?

Shiny new Docker Container versions:

- New esmero-php with PHP FPM 7.3.7 (with new tag) and also an extra docker-compose file made for people that like to code/debug using xdebug. This includes now also Exifinfo and Fido to extract pronom for files

- Cantaloupe IIIF server 4.1.5 with updated configs, processors. There is a huge Speed improvement and also sharper images when working on a Cloud Deployment using AWS S3 instead of local Mini.io (chunking)

New Drupal Deployment version (8.8.2, as new as it gets!), Tested associated modules we provide, including some Patches we submitted to that awesome community. We also got out first Drupal/Devel patch merged (so cool!) 

- Update Solr Configs running 7.5

- New new Min.io. More stable, faster

- Config/Sync that has less modules (we removed some un maintained ones and those who are not catching up with our needs) but more stable ones too. Everything was tested. Actually testing contributed modules took us longer that Archipelago itself.

- Config/Sync includes Field Formatters (How you see an Object) and View Modes for all our new Features, so you can test directly.

- Leaflet Maps in every Archipelago Digital Object  (ADO)

- Cached JSONLD files for schema.org. Because Google forgot to pay hosting and we had an embarrassing moment in December because of that. So now we have a cached backup. That if deleted gets updated auto-magically by Stawberryfield.

What is new in Archipelago?

New things (some of them you may already know, others for sure not)

- All formatters now scale to 100% width (we provide this setup by default, you can change it of course) , and allow proportional height.

- 3D Formatter is smarter now. Uses extra JS and has a tiny spinner so you know its loading. Can load .STL and OBJ and there is already stub code to load from the same Object Textures. So your thunder-cats look more 3-D!

- PDF Viewer. Using core pdf.js from Mozilla. Super simplistic, Render: Pages forth and Back. Reason for going this way is not because we are cheap (or lazy) but because Mozilla asks people NOT to use their fancy one. We respect other OpenSource projects, so we will extend our implementation to allow for more features in the next release but without copy pasting all their hard work.

- Better logic in our Metadata Displays (you won't see it but its there). You can tag what is the expected output of a Twig template. Also the Entity is self-rendering and deals now way better with Cache tags. Basically, its faster.

- Mirador Viewer 3! Using the latest Beta. This one is awesome. It can provide resources from 3 different sources, the ADO itself, referenced ADOs (like ismemberof, similarto, etc, etc, fields) and even other external IIIF manifests referenced in the Strawberryfield metadata.You can even decide which is primary and gets the first Space in the viewer. All is driven by our next feature, exposed Metadata endpoints.

- Metadata Endpoints. This was in progress last time you read about Archipelago but now its robust and works quite good. You can expose any number of Endpoints on each ADO using the same Metadata Display Twig templates that drive IIIF, MODS, etc. in the form or /do/uuid/metadata/iiifv2 etc. /do/uuid is a special resolver we added to Drupal so any entity can be exposed to the world via UUID. This will give us the chance of doing better things in the future regarding sub-functionality on each ADO.

Cache tags work there too (so fast) and basically makes archipelago any schema or ontology compliant. You just need to write a new or adapt one of the existing Twig templates to deal with the desired schema. By default you get iiifv2 and iiifv3 manifests (Mirador can use both). You can have as many as you want of course. To show how nice this feature is we added also a GeoJson endpoint with its super simple twig template.  And guess what, it feeds the maps. (our next feature)

- Leaflet based Map. With some cool setting options.  You can decide zoom levels, which Metadata Endpoint provides the GeoJSON features, same as Mirador, other 2 sources to draw related objects or other external ones if you want. (you can even hardcode some polygons into the geojson twig template and connect dots.. just for fun). You can even change on the settings directly what Tiles you want. Want some commercial fancy ones? Get a key and paste the URL there. We provide by default openstreetmap ones.

- Panorama Tours. New Metadata format to define complex tours. Jump from the USA to Jordan across the desert with a Click. Add hotspots that are other ADOs and they(should because we patched Drupal's lazy JS loader) render correctly on a popup. Or just add info.The same Viewer can deal with a tour or single panoramas. Tours use other Single panoramas, so are basically a Media-Less ADO. 

- Webform elements:

  * Nominatim with Ajax driven place to coordinates resolving. You type, you press search, you select the one that matches your desired place. Reverse resolving is in code already but not exposed publicly yet (leaving things to beta3 is needed!)

  * A Panorama Tour Builder Element. Like a full app on a Form element. Add Panoramas, select its initial rotation, add Hotspots, connect them, save (and after like days of hitting walls it really saves...).

  * Exif preview on uploaded Images. Works on local uploads and S3 too. This can feed also other elements, for now, you get a preview also of the image via IIIF.

- All media is exposed via a IIIF endpoint, even if API 3.0 still does not support it, means even Audio. This obscures also storage from endusers and allows us to safely provide access to files via their UUID but also conditionally to who is using them. (like if not attached to an ADO anonymous user can not access them! Safe!)

- New Mime Types to deal with formats Drupal knows nothing of.

- All our JS was cleaned and should be more reliable now (we have a lot of JS to bind libraries) which leads to: we load all JS from CDN. Faster on life sites

- New Key Name providers for Strawberryfields (SBF) , the cables that connect the RICH JSON to Drupal, so Drupal can see what is inside the JSON: JMESPATH. Means you can tell Archipelago to select one or more JMESPATH expressions from SBF and expose them to Solr or Drupal as a single Key. E.g All WIKIDATA elements and all the LoC Elements can go into "subjects" even if they are super nested. Means also you can aggregate diverse, strange, colorful metadata elements into a single property. Its like Xmas! See http://jmespath.org. Your deployment comes with some already setup so you can see how that works. SBF is still super smart. No values for a JMESPATH query? hu? No fear. It will always answer with at least an empty list. So you can even plan ahead before having the actual metadata.

- Speaking about metadata: SBF JSON's are more verbose. We keep track of which elements point to other ADOs or nodes and classification of Media and assets works by using the first component of the mime type. Means images go into a single place and are expanded to all the info you need, Documents, Models, Audio, Video, etc. 

- Auto Title Setter. Want to hide the ugly Drupal Title field when inputting? Did you know the Title is needed? But Drupal allows you to hide it just to die on you when not there. We have the cure. We take the webform "label" generated JSON key and make it the ADO Title. What if none is there? We give it a generic name so you can correct. But wait.. why? Well Drupal's titles are quite short. So we allow super long, UTF8 complex titles to live inside the metadata too. And Node Title gets, natural-language truncated when doing the auto update. Also, this is driven by and event, so can be extended by developers.

- More events. Yes, You can code (if you want, no need really) to react to more stuff on Metadata and ADO CRUD operations.

- Type to View Mode mapping. This is also cool. Previously you, human, had to decide how a given Digital Object should be displayed. Given a set of View Modes (we provide a BUNCH ...don't know what a view mode is? well, its a set of configs that decide how your metadata blooms into different viewers, metadata blocks, displays, etc.!) you had to select during creation or editing. Now, if you don't, or your Admin is hidding that option from you, based on the value of JSON "type" key (book, Podcast, 3DModel, etc, etc) we trigger your desired View Mode. You created new View Modes? You can add to the mapping. Good thing is, we also allow people to force one view mode (like Podcast v/s Podcast with Video, PDF and Zine!) Both share the same type, so you can decide what works better for you.

- Naming conventions. While setting up View Modes you get 4 Strawberryfields that you can format in different ways. AS you know, an ADO is really a single Strawberryfield full of JSON. So why 4? Well, its really just one with 3 virtual clones (one single query to the DB for the performance savy users). What is nice now is that all 4 are named Strawberry using an EMOJI of a Strawberry (so you know exactly where the fun is) and also the label itself says strawberry in different human languages, including Navaho =)

- Centralized IIIF settings. You get central settings for your public IIIF server and your internal one too. Still, you can override this per Formatter/Viewer. So you could even have a super performant IIIF server somewhere else just for your 40K images. Flexibility.

- Deposit 

- Solr based Entity Data Views (sounds fancy right?) basically some deep code that allows Solr to drive Data queries that return Entity lists. Like, get me all ADOs of type book, with Subject "Poetry". These can be used to drive autocompletes (Drupal does this only via SQL so this is super nice for us, also because the queries can go deep into the JSON metadata). 

- Simple one: Upload any files into an ADO. Archipelago will try to sort them by name (natural order, or numeric sequential) for you and give them sequence numbers in the metadata. You won't even notice it (trust me) but book pages will appear in the right order without you even asking for!

Previous Features:

All still there of course. (See Beta1)

Fixes:

- Too many to list. But in general code is more robust and smarter. JS is less. Schemas (how our configs are saved in YAML files) grew and had to be fixed. That was really hard. Like gosh it was hard.

- We have of course identified things that can be better, and we hope you all step up and tell us what else can be done, point us to any bugs so we can fix them, break this and make as many questions as needed.

TODO:

- A lot too (see issues in each repo). Beta3 will go out quickly (1 - 2 months) and V1.0.0 will come after that immediately. Core functionality is here already and we are now thinking of better UX/UI and heavy use cases (like Webarchives, HUGE, AI, file uploads and automatic recreation of an archipelago from S3 Storage without human, dog  or cat intervention. See Roadmap https://github.com/esmero/archipelago-deployment/issues/35 (feel free to ask for stuff there, 311-REPO, happy to make someone else happy)

Ok, uff..i'm tired of typing and pretty sure you of reading, i also promised myself i would not work this weekend, well, gosh, good try Diego. Surely i missed things and mistyped a lot. Its hard to speak about all code-things and the effort that is behind a beta release without acknowledging the actual people that made this possible, their time, work, patience or those that just believing that we can have a more empathic, fair, well planned repository system gave us strength to keep on going. Let's start. Sincere thanks/ gracias, Team, local, remote, national and international: Marlo, Mitchell, Giancarlo, Nate, and everyone at Metro. Obrigado to all the ones that have listen to my too many and too long conversations about this, thanks to our amazing Advisory Board that has brought us many times to stable ground and gave us sincere good advice, Danke to all the stakeholders believing in this project including some wonderful WIP implementations like: the Oral Histories Meta-meta modeling project by Natalie (you-so-awesome); Virtual Panorama Tours/ESIE from Upstate by Sheryl, Allison =) Ken and team, Finding AIDS with Jen and Zach, and so many others. I feel i lost the thank-you mojo this time, but we have you all in our hearts and we appreciate any and all contributions so far and to come. That said, we really want more involvement. Don't be shy. Nothing here bites nor haunts you, no interaction forces you to give up your other projects and systems. Its a space to explore.

Hope you can find something in here that connects Archipelago and its vision with your work, needs, and hopes and also space to imagine new use cases and "ways" with the tools we wrote with care for you. If not, at least you made it so far into this release post

Best and good night.

Diego Pino

Architect/Do-er/Archipelago

Assistant Director for Digital Strategy / Metro.org

Live versions (code.. look ma, we made a lot of commits)

Beta 2: 

- https://github.com/esmero/format_strawberryfield/tree/8.x-1.0-beta2

- https://github.com/esmero/strawberryfield/tree/8.x-1.0-beta2

- https://github.com/esmero/webform_strawberryfield/tree/8.x-1.0-beta2