how to install islandora software on a large scale?

[Jaafar Habibi]

We have 400 universities in our country. Each university has different faculties and therefore different specific libraries.

How should we deploy islandora software on the above-mentioned structure? How should be the architecture of the installation?

[Mark Jordan]

Hello, thank you for asking about Islandora.

Five options I can think of are:

1. one instance is shared by all universities; in this case, each university could have its own top-level collection and each library could have one or more subcollections within the university's top-level collection
2. each university has a single instance, which is then used by all libraries within a university (again, with each library having its own top-level collection containing subcollections)
3. each library has its own independent Islandora instance
4. groups of universities (perhaps based on type of university, region, language, etc.) could each share a single instance
   
5. some combination of the above four options
   

I am not aware of any country-level Islandora installations but others on this email list might be. There are a number of large shared Islandora instances, either at the institution/university level or shared by a group of independent but allied institutions.

So, the answer to your question depends on how much universities, or libraries within universities, want to share infrastructure, support, branding, and metadata/search functionalities. I think this is the set of questions that should drive the technical architecture of hosting Islandora.

With regard to the technical deployment, Islandora 7 (which is approaching the end of its supported life cycle) had a mature feature called "multisite" which enables any combination of the above architectures. Islandora 8 (the version strongly recommended for new installations) will be capable of the equivalent of multisite but the community is still working to understand the best ways to achieve that; this group actively working group working on multitenancy for Islandora 8. Currently, Islandora 8 has two robust deployment toolsets that can be used to implement any of the options above, a Docker-based toolset and an Ansible-based toolset.

Feel free to ask for clarification or examples, we strive to be a very welcoming community!

Mark

---

From: island...@googlegroups.com <island...@googlegroups.com> on behalf of Jaafar Habibi <jaa...@gmail.com>
Sent: Thursday, August 13, 2020 6:45 AM
To: islandora-dev
Subject: [islandora-dev] how to install islandora software on a large scale?

 

We have 400 universities in our country. Each university has different faculties and therefore different specific libraries.

How should we deploy islandora software on the above-mentioned structure? How should be the architecture of the installation?

--
You received this message because you are subscribed to the Google Groups "islandora-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora-dev/dc11f6df-49cf-44d6-b724-7452e7b27a69o%40googlegroups.com.

[Diego Pino]

Hi Jaafar, having installed and managed too many islandora 7 (and Drupal 8 instances) in my life from small to large ones, under real hardware, Virtualized one, Clouded one and a country level one when i was back in Chile (not too many but we had from the Atacama desert to the cold southern patagonia, a pretty cool for that time ensemble of university repos ) i would suggest  please do not plan on doing 400 institutions before learning/knowing and testing managing a single or a few single ones first. Backend scaling/hardware and deployment is not everything, and actually user admin/managing/metadata and media workflows and ingest, how things look! and how different the needs are between users will define a lot of your decisions. Islandora is a good platform but its always good to know what you are getting yourself into (and the 400 group of people too)

Mark's suggestions are perfectly correct but i would personally not go for 1. in your use case. Because pretty sure it will be just too much data and too much of an SQL query burden for Drupal 8, its just too much and if Islandora 7 most likely all your $$ will go into feeding memory to Java and still it will be never enough. E.g in Islandora 7, one of the repos we run has 469000 objects in a single Hardware/machine ensemble on Islandora 7, and comparing with the larger ones this community manages it is not that much and still  so, when i have more than a few admins at the same time things get complex. quite complex, and when derivatives or ingest happens we all cross fingers. It is how this work, there is a lot of human effort behind.

We have tested options 2 and 3 from Mark's list and i can confirm that can be scaleable and a better plan for you, with the added benefit that you can have large scale inter repository Solr indexing/discovery (also tested and works quite fine) so you could have cross repositories search. 

Sharing instances between different institutions is like sharing a toothbrush, eventually it will get messy, some will use more and affect the others and yeah, not healthy. In other words you will have different needs, some will scale too much, other fall behind, some will need customized views/look/theme. Not that great to be honest and larger/more popular ones will make the smaller ones have to pay for speed while they do use more of the resources so my recommendation based on experience and too many sleepless nights is split and distribute as much as you need/can and test, test test before going any of the ways. Also in software architecture there is not so much you can scale vertically, horizontally allows you to escape fixed limitations like storage I/O, number of open files, http constrains, etc. even universities deciding they do not want to be part anymore and go for other platform.

All this said (so you measure how much you take my opinion in consideration) owe only run now in production Islandora 7 and all new projects now into a different Open Source Platform we built, that means my opinion is tainted and will be different to other's invested in this current roadmap, which can be bad or good, but its an honest one. Islandora is a loving community and for sure you will get a lot of help in your discovery/testing process.

Good luck!

Diego Pino

[jaa...@gmail.com]

dear  Mark Jordan   and dear  Diego Pino thanks for your help and support.

I found your guidance very useful.

[Diego Pino]

You are very welcome Jaafar. Please don't hesitate on following up with any additional questions regarding your use case, pretty sure the whole community would love to hear what your plans are and also will be more than happy to help you out. 

Have a nice weekend

Diego Pino Navarro

Assistant Director for Digital Strategy
Metropolitan New York Library Council

599 11th Av. New York, NY 10036

{

  "Archipelago Commons": [

   { "url": "https://play.archipelago.nyc" },

   { "url": "https://github.com/esmero" },

   { "url": "https://groups.google.com/forum/#!forum/archipelago-commons" }

  ]

}

[Drazenko Celjak]

Hi Jaafar,

in Croatia we built a national repository platform DABAR for HiEd &
research institutions based on Drupal/Islandora 7 Multisite installation
and at the moment we have 142 repositories / virtual hosts with 128k
objects: publications, audiovisual content, learning objects, research data.
Our installation is highly customized: connected to various parts of
national e-infrastructure like Registry of scientists, Registry of study
programs, HiEd Information System, Portal of scientific journals...and
it will be connected to the national CRIS system which is currently
being built.
As for the hardware, we use virtualization and have different components
(Drupal, Fedora,...) on separate VMs. System like this needs constant
monitoring and tuning. We also gathered a community around our
repository infrastructure and we are trying to reach (national)
consensus regarding new features and metadata descriptions (time
consuming but good for interoperability and ease of maintenance).

Regards,
Draženko Celjak
SRCE - Unversity of Zagreb, University Computing Centre
www.srce.unizg.hr/en

> --
> You received this message because you are subscribed to the Google
> Groups "islandora-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora-de...@googlegroups.com

> <mailto:islandora-de...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora-dev/dc11f6df-49cf-44d6-b724-7452e7b27a69o%40googlegroups.com

> <https://groups.google.com/d/msgid/islandora-dev/dc11f6df-49cf-44d6-b724-7452e7b27a69o%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Jaafar Habibi]

dear   Drazenko Celjak     thanks for your help and support.

You received this message because you are subscribed to a topic in the Google Groups "islandora-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/islandora-dev/VdmbUSqIJo8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to islandora-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora-dev/7a5c9832-653b-8809-cbfa-b2d7ed161065%40gmail.com.