Archipelago 1.1.0 release whisper (like a warm breeze that slips kindly, unnoticed, through tall green summer grass)

50 views
Skip to first unread message

dp...@metro.org

unread,
Jun 28, 2023, 4:12:20 PM6/28/23
to archipelago commons
Dearest community members, critical thinkers, friends, bystanders and observers,

We are very happy to announce that all code (Modules) are wrapped up for this 1.1.0 release. It has been a winding (vertically and horizontally, x,y,z) journey since 1.0.0, one you (or i) would love to repeat, retrace a few more times in this timeline we are lucky enough to share making maybe different stops to observe the missed scenery and enjoy (or value more) the company of one or more of you friends that walked with us under this, our common, sheltering sky. Releases are a bit like that. You go back and reflect, observe, look and try to re-visit decisions, words, phrases, ideas and the many whys and the many "for whom" that drove what was built (the path walked is the mental construction, the places visited were already there). Even in a cold environment like code writing and software architecting there are many emotions, feelings and even /* tiny jokes */ (one must laugh under a deprecating technological world in-between Drupal updates) embedded in our communal output. And thus, i feel proud and touched by this release and by the many of you that have accompanied, joined and advised us on these travels.

Is Archipelago perfect? (one must question this many times before and after one's morning coffee). No. Perfect is a personal construct, not a communal concept. But i have to share yet again, that all our initial hopes, architectural decisions and envisions built 5 years ago (the ideas - pre-code -) under a stormy big sky in Montana are still true and we have not deprecated/forced data/metadata changes or anything core to your and our concerns (nor will) during this time. Archipelagos have been running steadily for all this time (first one is about to be 5 years old-young by the end of this year!) without any larger changes in your/their precious digital objects and the implementations and community has grown (adulting?). But that does not mean we have not evolved our ways, improved and critically adapted our approaches, corrected mistakes, made our ears and eyes (hearing and vision) sharper to community needs and, brought, through real use cases and our own team's experience, an (absurd) amount of new features and enhancements.

In concrete numbers this release brings 16,925 new lines of code (with some underlying features i will highlight in the next paragraph, keep swimming) and we removed 1,539 lines (no comment was hurt). And all of this is just actual code. Not counting YML files and config files or the new module(s). I wish these were different numbers (the adding, not the removing). We have a new Module named Fragaria (dynamic redirects using any path/structure you want and using Indexed Solr values) and 3 new submodules. The name for the former comes from https://es.wikipedia.org/wiki/Fragaria_chiloensis. My favorite Pull request message was "I'm so full of ty*pos" which will apply to this message for sure too. I think we also brewed circa 37 kg (metric ok?) of dark roasted coffee during this release -many cups went cold but were consumed anyways -. Ok. This is where i stop being cold and start to thank you individually: 
Allison, your first code commits (AMI permissions and LoD sorting) are in this release. Thanks for all the managing, testing, working, re-using, proposing, pushing back, recycling and the long code discussions with me (most pull/merges went through hours of screen sharing), new features, opinions, code and literally millions of spreadsheet rows + site bulding and templating. Thanks for everything you keep bringing (soul, professionalism, care) to our community. Our internal team: Katie for working with implementations, migrations and LoD, learning (the hard way) the Archipelago ways, the meetings, using and breaking the tools and to Albert for your contributions to ISSUEs, code, dev-ops, fixes, community support, needs and use cases. Thanks team, this is hard work and always appreciated.. Nate, our big boss for believing, supporting and pushing this project under METRO's goals and future plans, allowing us to be OSS and also fully long term sustainable (normally an uncommon mix) and in general thanks to METRO's staff that had (and has) to (sorry folks!) hear about the "amazing" things we did/added/removed every Monday morning during our staff calls. Thanks to every vendor in our community for testing, implementing, devoting their staff’s time, extending and breaking Archipelago and in particular to Pat for your code contributions, the OAI module and for keeping up with my code reviews and the esoteric debugging experiences shared. To the IIIF Community for pushing our specs and needs, for all their work, use cases and lovely community. To everyone (So many institutions and individuals. I can't mention all of you anymore!) for using archipelago and trusting your loved PDFs and Images and Videos, and Sounds and 3Ds and Maps and etc. to it, adding fun emojis on slack, asking, requesting, suggesting, breaking, reading our many strange (mostly mine are strange) ways of communicating, adapting and helping to make everything awesome, better, good, human. 
Thanks to the good friends we made on this road, dearest Mike, Johannes, Alessandro, Jen, Brenden. To our workin group. To the ones that started (and inspired) everything with us (Giancarlo caro amico, grazie mille!). Gracias. Danke

Highlights:

- Fully revamped and expanded seach (Solr) experience. Allowing Aggregations of Flavors (OCR, Website extractions, etc), Advanced Search. So much JS with many last minute improvements and Solr JOINS, custom filters and query alters, custom Smart/fast Highlights (making subqueries to get around limits of Solr) basically touching every piece of the Search API to get around our massive needs (hundred of thousands of OCR and NLP extracted entities on 16 Gbytes of RAM!)
- Better Formatters, better settings and absolutely new ways of making viewers, through IIIF and GeoJSON interact with each other. Flip a page (canvas) and the map can (and will) react, a View will refresh and requery Solr, etc. IA Book Reader has now better Mobile support. New Formatters like the Citation one and Twig extensions too! So much for IIIF/Mirador. OSD has geo annotations now and can also parse IIIF Manifests (PDFS and Annotations on OSD!) and Leaflet can deform/transpose images from a Manifest + GeoJSON at the same time (you can map middle earth to Ireland now if you want).  Replaced the extra custom MIRADOR with a better one I built (JS).
- Smart caching on Twig templates/Metadata Display Entities + eye candy (preview/save follows you around). Does your template call/render a view? We got you. We will cache the cache tags (yes it is a thing) and invalidate all down (inclussive other templates used in the view you are using). Also Solr indexing of Child ADOs invalidates parent ADOs now (no more stale IIIF Manifests)
- Twig processing on JSON API Calls so you can feed your Repositories weirdly shaped JSON without ever using the UI. New control metadata tags (skip Post processing, etc)
- Composting of old/temp files and smarter cleanup logic via queues.
- Better Background Processing
- AMI. SO much! Logging to file, better LoD Reconciliation, better permissions. Safer.
- Views Extensions and improvements. 
- Ajax URL Browser state (we even replaced the Core Views Ajax routes/libaries with smarter ones) plus TONS of custom JS to deal with Facets/Views/Blocks.
- A whole Facet ecosystem, e.g getting around the issue of "no facet summary" when no results and new Facet Widgets + Options (remove/rename values)
- Better remote filename/mime/extension managing. Even if the file comes from a weird place.
- Remote file OCR/XML/Transcript/VTT and even JSON full text extractions
- Better smarter NLP processing with fasttext language detection
- You

Of course there is tons more. And also things (many) you will never notice (which is good). You can not even imagine and I can barely remember. Because of that, the full release Notes/logs are at the end.

The actual Demo/deployment strategies repositories (new Solr 9.1, docker variables, new theme, new settings, new Views and new templates) should be ready to be shared by Friday night. That is a clockwork/surgery type of effort and it will take us some extra night time(s). Thanks again (truly). We really like you all. 

Questions? Please make them? How to update? On my next (Friday message, but also it is a single docker exec line soooo maybe this works for you? 'docker exec -ti esmero-php bash -c "composer require strawberryfield/strawberryfield:1.1.0.x-dev strawberryfield/webform_strawberryfield:1.1.0.x-dev strawberryfield/format_strawberryfield:1.1.0.x-dev strawberryfield/strawberry_runners:0.5.0.x-dev archipelago/ami:0.5.0.x-dev drupal/google_api_client:^4 monolog/monolog:^2.9"'

Documentation branches? For sure by Monday. Working on it.

Hey. Thanks again. We had a blast. Let's travel together again. Happy to carry your heavier bags and books. Maybe share a snack and a long talk too.

RELEASE NOTES/LOG:

1.1.0 Release of Strawberryfield (v1.1.0) Latest

What's Changed

ISSUE-234: Make Flavor search aware of CWS/Children based OCR by @DiegoPino in #235
ISSUE-240: Improve compost times and allow getTempDirectory() as safe path by @DiegoPino in #241
ISSUE-242: adds more data to Flavor Search result JSON by @DiegoPino in #243
ISSUE-244: Adds Flavor aggregation at Parent ADO Level (2 levels up) by @DiegoPino in #245
ISSUE-247: Fix bug on parent of parent and cleanup empties for legacy OCR by @DiegoPino in #248
ISSUE-249: Fixes wrong use of static:: by @DiegoPino in #250
ISSUE-251: Simulate message in StrawberryfieldJsonPatch in wrong place by @aksm in #252
ISSUE-253 + a bit of 246: New Highlight processor + Textarea for JMESPATHS by @DiegoPino in #254
ISSUE-261: Adds ap:tasks title source option by @DiegoPino in #262
ISSUE-257: Humble (and experimental) attempt at dealing with stale/missing excerpts by @DiegoPino in #258
ISSUE-265: Allow a workaround for dot files when composting by @DiegoPino in #266
ISSUE-263: Improve HL for JOINS + Phrase Linking. by @DiegoPino in #264
ISSUE-268: Modern JS (latest 112.0.5615.121 (Official Build) (x86_64)) and our wonky #states bug by @aksm in #269
ISSUE-270: Enable the shortest (third option) Path for breadcrumbs by @DiegoPino in #271


Format Strawberryfield 1.1.0 (v1.1.0) Latest

What's Changed

ISSUE-221 --- ADO Type condition plugin. by @patdunlavey in #222
ISSUE-128: Basic pass custom Mirador 3.3.0 with Image tools by @DiegoPino in #235
ISSUE-237: bad indentation for UV formatter schema by @DiegoPino in #239
ISSUE-236: Citation Formatter check for null by @aksm in #238
ISSUE-241: Citation Formatter settings undefined variables by @aksm in #243
ISSUE-244 --- Workaround Mirador to allow mutiple Audios on Chrome/Firefox by @patdunlavey in #245
ISSUE-249 --- removed '#required' from the 'ado_types' field in the ADOType condition configuration form by @patdunlavey in #250
ISSUE-247: Fixes Search API twig extension by @DiegoPino in #251
ISSUE-262: Internet Archive Bookreader mobile menu is broken by @aksm in #263
ISSUE-167: Style "share" pop-up window for IA Bookreader by @aksm in #264
ISSUE-223: Provide media/files at UUIDs additionally to node IDs by @aksm in #268
ISSUE-260: Views and Facets + advanced search by @DiegoPino in #261
ISSUE-269: Lower the strict conditions for a single Media (audio and video) with VTT(s) by @DiegoPino in #270
ISSUE-269b:Fix height on VTTs/Audio by @DiegoPino in #272
ISSUE-271: Deal with Empty Queries on Advanced and Facet Summary for Queries by @DiegoPino in #273
Fix Issue 276: advanced search add/remove without exposed operator by @patdunlavey in #278
New submodule: format_strawberryfield_rest_oai_pmh by @patdunlavey in #277
ISSUE-280:Exclude Facet Processor that allows multiline facet value settings by @DiegoPino in #281
ISSUE-280b:Trimming was needed. But we want to preserve spaces at least by @DiegoPino in #282
ISSUE-284: Uppercase the Conjunction Operator to please the Solr Gods (rey sol) by @DiegoPino in #285
ISSUE-265: Allow multiple CSS classes for clipboard copy functionality by @aksm in #266
ISSUE-283:Facet Summary Processor misbehavior on Date Range when it is the only Facet selected by @DiegoPino in #286
ISSUE-274: Report render array output error in Metadata Display preview by @aksm in #275
ISSUE-287: Adds a new JOIN filter to allow Flavors to matching ADO unified search by @DiegoPino in #288
ISSUE-290: Solr 9.x and better HL for joins and Advanced search JOIN awareness by @DiegoPino in #291
ISSUE-290b: Remove op query parameter from the Browser state by @DiegoPino in #294
ISSUE-298: Modern JS (latest 112.0.5615.121 (Official Build) (x86_64)) and our wonky #states bug by @aksm in #299
ISSUE-300: Make UUID Entity Reference act on String based defaults (UUID and Arrays of UUIDs) by @DiegoPino in #301
ISSUE-289: Misreported unused JSON keys for multilevel keys in Metadata Display preview by @aksm in #296
ISSUE-304 --- Use the correct variable name for the aggregated fields by @patdunlavey in #305
ISSUE-308: Fix null coalescing operator's order at DateRangeProcessor.php by @DiegoPino in #309
ISSUE-310: Show Pretty printed JSON when rendering using native/validated option by @DiegoPino in #311
ISSUE-302: A Saga of JS driven IIIF Interactions by @DiegoPino in #303
ISSUE-313: Save button on Metadata Display Preview on default theme is under Code Mirror by @DiegoPino in #314
ISSUE-315: Twig nodes accessor to non-existing name attribute throws exception by @DiegoPino in #316
ISSUE-310: lazy me, Update MetadataDisplayForm.php by @DiegoPino in #317
ISSUE-320: Disable Variables and Custom Error handler from Twig preview until more tests have been done by @DiegoPino in #321
ISSUE-323: Layout Builder uses a totally different Field UI render array by @DiegoPino in #324
ISSUE-322: Better related entities/configs cache tag calculation for Metadata Displays by @DiegoPino in #325
ISSUE-326: Advanced search community needs by @DiegoPino in #327

Webform Strawberryfield v1.1.0 Latest

What's Changed

ISSUE-145: Give rdftypes and vocabs defaults for everything on MultiAgent Element by @DiegoPino in #146
ISSUE-147: Wrong use of static:: in WebformPanoramaTour by @aksm in #148
ISSUE-149: Fix Nominatim Controller Server Exception log by @DiegoPino in #150
Issue-151: Local Options, CSV and JSON autocomplete for LoD Vocabs by @DiegoPino in #152
ISSUE-153: Solr driven entity autocomplete can not use match operator by @DiegoPino in #154
ISSUE-155: Fixes Objects (single value entry for an webform element) being misidentified by @DiegoPino in #156
ISSUE-157: Fix Autocomplete deprecation (search api 1.24+) by @DiegoPino in #158



AMI v0.5.0 for Strawberryfield 1.1.0 Latest

What's Changed

ISSUE-124: VBO breaks after 4.1.2. Update composer.json in 0.5.0 by @DiegoPino in #126
ISSUE-8: Add CSV reporting + Skip on missing file(s) by @DiegoPino in #123
ISSUE-129: Better remote Filename handling and small improvements by @DiegoPino in #130
ISSUE-131: Fix wrong logic. Status might not be preset (yet) by @DiegoPino in #132
ISSUE-134: Fix AmiStrawberryfieldEventPresaveSubscriberJsonTransform priority by @DiegoPino in #135
ISSUE-136: Adds '#description_display' => 'before' to avoid warning by @DiegoPino in #137
ISSUE-138: Caps filename and uri to Drupal's File Entity build in limits by @DiegoPino in #139
ISSUE-145: Multi Relationship improvement by @DiegoPino in #146
Add sort for csv headers on LoD Rec form by @alliomeria in #150
AMI Access Control Fixes by @alliomeria in #152
ISSUE-153: data_lod_contextual that includes source labels by @DiegoPino in #154
ISSUE-156: Owning user deletion and errors on default formatters (and others) by @aksm in #157
ISSUE-159: Remove spaces and dots from the beginning in any combination for remote filenames by @DiegoPino in #160
ISSUE-162: Modern JS (latest 112.0.5615.121 (Official Build) (x86_64)) and our wonky #states bug by @aksm in #163
ISSUE-164:Change master request from symfony (was never there) to the shim by @DiegoPino in #167
ISSUE-161: AMI CSV export fails when an ADO has a missing type key by @aksm in #168
ISSUE-171: Check for AMI existence. In case it was delete or exception log to DB by @DiegoPino in #172


Strawberry Runners v0.5.0 for Strawberryfield 1.1.0 Latest

What's Changed

ISSUE-59 --- Fix fatal error when \Drupal\strawberry_runners\Web64\Nlp\NlpClient::post_call is called with empty params by @patdunlavey in #60
ISSUE-57: VBO action + Pure text extraction from "anything" + File based HOCR with Image matching by @DiegoPino in #64
ISSUE-66: Allow ap:nopost to skip processor IDs (machine name) by @DiegoPino in #67
ISSUE-70: Improves process logic and cleans up atomic/processor generated file/garbage by @DiegoPino in #71
ISSUE-77: Fix binary detection for Text extraction by @DiegoPino in #78
ISSUE-79: Use flv:exif as fallback for Image to HOCR matching by @DiegoPino in #80
Full Changelog: v0.4.0...v0.5.0

Fragaria 0.1.0 Release Latest
First release!

On behalf of our core team
Abrazos + Hugs (the same but not really the same)

Diego Pino Navarro
Director of Digital Strategy
Archipelago architect
Metropolitan New York Library Council
PO Box 2084
New York, NY 10108 

Nate Hill

unread,
Jun 28, 2023, 5:06:20 PM6/28/23
to dp...@metro.org, archipelago commons
Congrats to Diego, the METRO team (Diego, Allison, Al, Katie), all of our members and partners, and the entire Archipelago community on the new release.
I am so inspired by your work!
I want to also give a shoutout to DCMNY, the Digital Culture of Metropolitan New York. It was recently migrated to Archipelago and relaunched, and it looks really wonderful!
For me, this was the origin of Archipelago. 
DCMNY is a multi-tenant repository that METRO has offered as a service to our local members here in NYC and Westchester County for a very long time. It has seen many iterations and many platforms come and go.
5 years ago, when the poet who wrote the prior email presented the idea that the best path forward for DCMNY was rethinking repository architecture and starting a new open source software community, it seemed like a pretty risky choice.
It was the right choice!
Friends, please take a moment to explore DCMNY and the many interesting things there. If you are already a METRO member in our immediate region and you want to contribute collections to this New York themed repository, let us know. If you are elsewhere in the state / country / world and you happen to have collections about New York, we can accommodate that as well. If you happen to have a digital collections connection to NYC, this is a really lovely way to spend more time using Archipelago and to become part of a community of practitioners.
Have a good day or night, wherever you are
Nate


--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/9ff67662-402c-45e2-b6a6-7c724f785d43n%40googlegroups.com.


--
Nate Hill
Executive Director
Metropolitan New York Library Council
Reply all
Reply to author
Forward
0 new messages