Archipelago 8.x-1.0-beta3 day and our last silly release announcement beta before 1.0.0 version
82 views
Skip to first unread message
Diego Pino
Jul 17, 2020, 12:25:44 AM7/17/20
to archipela...@googlegroups.com
Good night dearest 87 members of this group, their families and pets.
What a strange 1/2 year. What a 0.5 year full of social contradiction, systemic dichotomies, inequalities, uncertainties, too many video calls (won't name any specific app here), to much screen time, too many times #bitcoin mentioned in social media, confusing messages from elected leaders, too much bread making, probably some fear, anxiety about the now and the probably future and what not, maybe even lack of sleep and eating too much (same baked bread)/too little because of this or also just because.
Feels too much even without talking about everyones health, body and mind, about the ones we will miss (sigh), the effects of this global pandemic on society, everyones lives affected in every corner of the world, the word 'normality' gone and global warming. Well it could be worse and i'm not here to complain, quite the opposite, I'm really here to acknowledge that even in complex situations like these, the many (but not too many) that had hopes for this tiny little dreamy pet project named Archipelago stepped up, helped, tested, broke code, cared, shared and trusted patiently, laughing at danger while helping to move this humble system, its weird philosophy, a few more steps further and we all managed to get another beta, hear me! the last beta release in the 1.0.0 series. And we/i are eternally thankful for that.
4097 new lines (minus two thousand or so i had to scrap and some tabbing and indenting that had to happen too) plus some extra 6000 configuration lines, literally hundreds of diet sodas and good and bad coffee, mean, per se, really not much (well ask my keyboard, has something to say probably) what actually matters is that:
Around a lot of new use cases, old never solved needs, all that deploying, using and abusing, teaching and learning, reading and secretly writing emails a small, strong, fearless and resilient community has been emerging. As diverse as the metadata Archipelago can deal with (strawberryfield really), as persistent as our S3 storage, as humble as our docker-compose deployment, dynamic and vibrant as our media displays are, as aware of the importance of preserving the past as the practice of Web Archiving can be, as respectful as Oral History practitioners are, as funny as the many /* inline comments that tell the story of this development */ and as concrete and future looking as the IIIF manifests we love, depend on and can edit in realtime.
I know its not a very verbose community (ha! c'mon you so shy) but its a loving bunch of people. All these new features, bug fixes and atlas obscura performance improvements, the many in-between-the-lines, future ideas, open issues and the gentle logging messages that say nice things like 'Thank you!' are dedicated to them (you). Also to the ones that still don't know about this software (poor people) but were/are in our minds while designing and the ones that refuse to embrace simplicity. We get you. Its hard to let go. I normally list people here at the end, but nobody want to read until the end. Thanks Nate, Mitchell, Davis, Mark ,Tracy, Shelly, Anne, Kyle for supporting this project from inside Metro and in specific Nate for giving us the space and helping even with CSS! Mitchell for being patient and not falling asleep during our daily calls. Marlo was with us until beta2 and contributed with some good code to the effort, thanks too. Sheryl and the whole ESIE/WNYLRC, Chris, Ken, Joe, Kathlyn, Heidi, pal Allison and everyone in the team for all the work on getting https://esie.space in shape and your great ideas. Extra special mention to Allison (2 mini twix bars as bonus), fast learner and metadata/descriptive-technical hacker and twig template wizard! Its so great to work with people that care. Tim Spindler from Long Island for your advocacy and help spreading the word (and your amazing deployment skills). Tessa, Jen and Zach for your public contributions, the trust and Finding AID challenges, The amazing folks of New York State Library Special Collections Department, Amy and PJ, you guys such fast learners and a joy to work with. Katie from SDSU for all the use case and space to talk about this as future option. All the energetic, positive and smart folks from our Advisory board, Josh (Reason you have IIIF manifests folks), Arnold for your wise advice and great energy, Esther, another special bonus for your kindness, encouragement and good ideas, Nadine for all your patience in this last year of migrations, Peter for all the experience you bring and the middle east work related to Archipelago, Zach(again, also best smile), Marii for all your deep reflections, devops concerns and platform ideas (leaflet!), Tim (again, wise Tim), Ben for adding context and scope/experience, Sumitra Webarchives super star, Natalie ,we so lucky yo have you, so much inspiring Oral Histories work and dear Henry for sharing all your experience. I hope i did not miss any one. Also to everyone that sends us positive energy, reads the messages, the posts, even from other platforms/OSS and our old communities. Ilya for your kindness sharing web-archiving replay code with us. You all good. Additional thanks to the great friends at Born Digital (Special mention to Derek this week, but not excluding ever Noah(natural leader), Pat (since my beginnings), Gavin(buddy), Sarah (high respect for your knowledge) and Hertzel (d8/9 wizard), you good partners and your work is always inspiring) and dear friend David Keiser-Clark for sharing objects with us. And finally, my partner, friend and most admired DEV and repo managers Giancarlo for all the testing, discussing, pulling, code reviews, merging and breaking. Gracias amigo for all your hard work and inspiring ideas. I forget people i know, there are more in my mind (Carolyne, Kim, Rosie, Brad, Brandon!) and past colleagues that were made immortal in the code. Almost crying here but can not confirm.
We believe repositories play a fundamental and important role in documenting, preserving and exposing "things", telling stories, narrating times, for future generations but also for current ones in media isolation. "Things" we humans care for, stories that require multiple perspectives to be fair and need to be described, cataloged and preserved in way that is respectful and faithful of the identity(ies) of their creators/observers/bystanders and witnesses. "Things" and concepts that need to survive strange times like 2020 or maybe 2022(hope not so soon). To avoid making the same mistakes (looking at you - racist subject headings) in the future. We are convinced Archipelago can help. We built it to play this role in a sustainable and socially aware manner and we built it incorporating as many voices as we could. We also made it economically sustainable, low carbon footprint.
Facts:
Deployment strategy can can be found at https://github.com/esmero/archipelago-deployment/tree/8.x-1.0-beta3
We know its not perfect and we have 2 weeks to make it better if you feel offended but the simplicity. But i feel it is more complete than Beta2 for sure (Beta1 was a nice one). We will keep code and configs open for 2 weeks max, fix(if needed) and then freeze. Also, you can change webforms and displays, part of the game, to make changes and modify.
Deployment Readme.md is updated to reflect the changes and the fact you can deploy demo objects now. Still takes 10 minutes or less? We kept the 5 Steps per OS (could be 1 click, no steps but we really want that people know what is happening, this is not whatsapp OK) plus a global optional one (made of 2 steps) for demo object deployment.
archipelago-deployment repo
Docker Containers updated mostly, but nothing crazy
- esmero-cantaloupe (cantaloupe IIIF server) at version 4.1.6 (latest) with our custom built
- esmero-php with PHP FPM 7.3.x (new tag) plus the additional docker-compose file made for people that like to code/debug using xdebug. This container contains exifinfo, tesseract, fido(pronom) etc.
- Speed improvement, /info.json aware for PDF pages (bug we found and superstar maintainer fixed) and sharper images when working on a Cloud Deployment using AWS S3 instead of local Mini.io (chunking), but chunking also works (and enabled) in Min.io. Please feel free to swap things around.
- New Drupal Deployment version (8.9.2, as new as one can be, like a few days old!), But wait for it.. We are on top-notch version of every contributed module! Check that composer.json. And hear me here. We are (since 2018) Drupal 9 ready.
- Update Solr Configs running 7.5 for latest Drupal Search API.
- New new Min.io. More stable, faster. EC2 deployment works quite well too and i saw even a Digital Ocean archipelago on 2 Gbytes of RAM running.. that is brave.
- More scripts, for updating code, we even deploy !menu items via JSON api now.
- Config/Sync that has even less modules (we removed some un maintained and other that were catching fire and throwing smoke). Everything was tested. 2x and 3x. As always testing contributed modules takes us longer that Archipelago itself.
- Config/Sync includes new Field Formatters (How you see an Object) and View Modes for all our new Features, so you can test directly.
-Demo Objects. https://github.com/esmero/archipelago-deployment (Feel free to correct metadata there if you need to). Contributions to more objects are always appreciated. you can use play.archipelago.nyc as play (yes, redundancy is my thing) ground to play and generate demo objects.
- And new displays and setup facets, more Drupal Solr driven views and such.
- New Metadata displays, including IIIF Presentation V2 and V3, some basic MODS 3.6, GEOJSON, SCHEMA.ORG, even a silly one for a carousel (so 90's i know) as showcase of all what you can do. IIIF Manifest V2.1 and V3.0 got extra logic and data, like max with per image based on EXIF and PDFs served page by page via IIIF.
- WARC using the amazing embedding machinery from Ilya from https://replayweb.page. A one in a lifetime collaboration which has led into organizing a web-archiving+archipelago IG.
- More Cached JSONLD files for schema.org. Because Google forgot to pay hosting (that was on beta2 but i can not forget) and now decided to deprecate like a lot of JSONLD files because yeah guys, that is the idea of linked data publish and then remove (facepalm). We had again some embarrassing moments. So now we have a cached backup and we serve our own too. If deleted gets updated auto-magically by Stawberryfield.
- We have now some experience with SSL deployment and have that automatic (5 archipelagos with SSL running live), will share docs about that in another post.
What is new in Archipelago? Please look at Beta2 and Beta1 if you want the full list. here only the fresh stuff
(anyone willing to ever take all my release notes (beta1) and (beta2) write a full features list? I invite the "refreshments" if so)
strawberryfield module
- Post processing Event driven Services. You get now File characterization, EXIF and FIDO via a plugin system and a Event Subscriber. Means also when doing batch ingest (what?) you get the same experience as with Webforms. And even some cleanup logic.
- The Concept of what a digital Object is as dynamic as it gets. We added bundle level permissions that move with you. Basically: attach to any Content type a Strawberryfield and it starts to work as a digital object including the weirdest one, a specific JSON API 'fake' file field, so you can deposit files without touching JSON remotely.
- New utility services for developers. Stuff like 'Is this bundle a digital Object?'.
- Strawberry-field can now decode/encode its own JSON. Value flattener is better.
- Drush Ingest. YES! So we use JSON API. I like APIs, But i know people don't like to write CURL calls manually and stuff. This Drush script (version 10) takes a JSON file (no strange stuff, simple pure metadata) and a folder with anything you put there. It uploads the files from the folders, classifies them, gets the UUIDS, rewrites the JSON, connects them, uploads the new Digital Object and persists. Ends "thanking you" when done.
- New CRUD events
- Bring Subjects Headings from LoC and WIKIDATA items and maybe a list of tags all together to Solr and name it 'subjects'. How? With the strawberry JMESPATH keyname provider you add a few jmespaths paths to your metadata (any deep hierarchy) and tell Drupal to use this as 'subjects' property for your strawberryfield. And you can facet now. No more XSLT, gesearch-es and stuff. All UI.
- We count milliseconds and memory use. Realtime performance tracking is a wish i had and its becoming reality. More abut that in a next episode, but gist is: we want archipelago to adapt/suggest what i can do in realtime based on your hardware platform.
- FULL Drupal 9. Audited and ready. 8.8.x || 9.x
format_strawberryfield module
- Webarchive Embed Widget! (see the demo object that we ship) and multi format, includes normal warc but also warcz collection format. This is all on Ilya Kreymer. He moved from 1.0.0 to 1.0.1 with so many new features in like 3 days. We also serve the code needed for the JS worker from a controller. Its actually simple and cool. Just upload your WARC to archipelago, give it the starting URL (don't forget to add the end-slash) and it will stream and replay the web in a viewer/frame. Mix and match with other resources. Next comes now Solr indexing and other metadata extraction and bookmarking. But this is just huge! clap, clap thanks Ilya.
- In general better code all around. We know we can do better, but we have more options that ever. Images, video, audio, books, pdfs, 3D, warcs, maps, IIIF viewers, template renders (like build your own viewer), etc. All Archipelago allows you to connect the RDF type (like any type key across all metadata) to the viewer, so basically objects can be setup manually to use snowflake mix of images pdfs and audio or automatic if 3D is present.
- Most JS CDN libraries got bumps and our own got better too (yes, we code JS too)
- Metadata can talk to formatters. Means you can feed settings calculated from your objects to theme and to Javascript (needed for Panoramas e.g) . Like what if you EXIF says you can not show a certain feature because the flash did not trigger while taking the picture? You could, not that you would, but you could. This adds a full extra level of configuration and flexibility options.
- More options for Maps. Like more zoom? less Zoom? other provider of tiles? you go for it.
- PDF Viewer had back and forth buttons inverted. Not anymore. Still almost monk like. no fancy nothing. Back and forth. that is all.
- Panorama Viewer JS is smarter and takes arguments from metadata)
- Panorama Tour Viewer is the same as the other one but with some crazy OOP logic.
- Metadata Displays (Twig) additionally to be self-rendering and having better Cache tags now also inject extra info for Twig template/metadata folks. Like you need to know if the user that is seeing things is logged in? Or you need access to the IIIF servers? Plus you can use Twig Bamboo and Twig Tweak (contributed modules enabled by default) to render views, blocks, other metadata displays, directly.
- Mirador Viewer 3 using the Release Candidate 3 (4 was released last night, had no time to test sorry) Awesome.As before you can provide resources from 3 different sources and even external IIIF manifests referenced in the Strawberryfield metadata.
- Metadata Endpoints are more robust and we expose a few more by default. Please don't stop, you can add more and they are faster. Want to share IIIF manifests between Archipelagos? connect one viewer to the other one's IIIF output and you have decoupled archipelagos.
- FULL Drupal 9. Audited and ready. 8.8.x || 9.x
webform_strawberryfield module
* Nominatim/Geographic reconciliation is better now. Some bug fixes!
* A Panorama Tour Builder Element is like a full app. Adding/removing hotspots, multiple panoramas.
* LoD remote api caller (autocomplete) is more flexible. VIAF is next. Hooking new ones is just a few lines of code! Also Wikidata with custom SPARQL is in my planning.
* Exif preview on uploaded Images with preview of the image via IIIF, inclusive if the file is still temporary.
* Library of Congress LoD in its full colorful extension. You can select which Auth. you want to use now. Or, you can ask for any response of type "Geographic", basically any type that LoC allows. PersonalNames? We got you
* You can hide the URL part of LoD autocomplete. let's say you are exposing the ingest form to non library/museum people. Only the autocomplete will be visible, URL will be stored hidden and then Curator can edit/change/if they want.
* Got a phone?Ipad, Galaxy? Doing Guerilla cataloging? You can upload video and pictures from your camera roll now.
* JP2 uploads work well in every browser (crazy that many don't recognize those as images)
* New Webform workflow. When you create a new Object, webform shows expanded. When you edit an existing one Webform is contracted and requires you to press 'Edit'. When that happens Node save button, state save, etc become invisible. So you don't get distracted or bothered by too many submit buttons. Finished? All appears again. You realize you just want to change from Draft to Published and don't need to go through the 10 steps of validation for every metadata piece? You can forcefully close the metadata form with a button (same as when finished) and all general node buttons re-appear.
* Finished and USPS got a package and you have to open the door and forget about the world? You uploaded files, cataloged for hours and computer shut down while chasing your baby bat around the house? No worries. Once you press Finish, the metadata gets saved in limbo. You come back to create an object in a day or later? We let you know there is an unsaved node and we load the data for you. This can also be made into a 'save draft' button.Also works for existing nodes and each UUID/ Node can keep its own copy. Once you persist/persist the temporary one gets purged.
* Auto Sorting/Self ordering of files based on their names became even smarter and now respect your manual order if you applied any. (proven in battle with books)
* Of course. XML upload/import. Yep.
Viewers:
- FULL Drupal 9. Audited and ready. 8.8.x || 9.x
Previous Features:
All gone. Kidding. (checks twice). All still there of course. (See Beta2 and Beta1 please)
Fixes:
- Too many to list. Help us by breaking stuff please. Deployment strategy can have issues because it is basically sitebuilding and then exporting. I apologize for all the pink color i used in the theme and the lack of contrast. I was feeling cute. But the theme now contains NO advertising (go to archipelago.nyc for that) but allows you to control many many color options, layout, etc, via the UI. Basically don't pay for fancy design if you don't need to/have no money for that. Pay to extend make mind blowing design if you have the resources of course.
@TODO:
- Again so much (FYI, i don't close normally issues, so an open github issue does not mean unresolved... it means look! Someone discussed this and if you read further you will see a merge and code examples. If not, ping me)
- So Please see issues in each repo. V1.0.0 comes in August. Main concern is ACL https://github.com/esmero/strawberryfield/issues/82. For IR needs. I want to make it well and fast and extendable. I will also make sure strawberry_runners (reactphp realtime background processing with queues and chocolate) is deployed in the first V1.0.0-RC. We will move out of 8.x nomenclature into 1.0.0 (semantic) and release will be of course 100% Drupal 9. We need more help. We will get there but we need more involvement.
Hope you can find in-between my bad dad jokes, bad english and silly sentimentalism something to relate to. I have tried to make every one i ever interacted regarding GLAM (and also biology/genomics, geology, paleo and some archeology) part of the code and the use cases. Hope Archipelago and how we think about metadata and future workflows resonates a little bit with your own work, If you are here you are basically crazy. Nobody reads my release notes completely. Pressing send without looking back (excuse my typos, hope i did not offend anyone)
Buenas Noches
Thanks!
Diego Pino
Architect/Care-taker-of/Archipelago
Assistant Director for Digital Strategy / Metro.org
All the code of Beta 3:
Nate Hill
Jul 17, 2020, 8:56:09 AM7/17/20
to Diego Pino, archipelago commons
Diego, not only is Archipelago a work of art, so are your release notes. Thank you for being you. This is lovely.
Nate
Diego PinoArchitect/Care-taker-of/ArchipelagoAssistant Director of Digital Initiatives / Metro.org
--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/1f741891-c43a-45f4-9240-b6373e87270co%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages