Asking for community opinion about Drupal 8 and Islandora 7.x-2.x "CLAW"

[Diego Pino]

Dear Community,

I wanted to share with you some ideas and also ask you for your opinion and feelings on the following topic:

Would it be a good idea to push efforts on our shinny new Islandora Claw project to develop for the upcoming (19 Nov) Drupal 8 instead of Drupal 7?

For those not involved yet in this new Islandora architecture, let me tell you what/how we are doing stuff right now (very simplified)

Our current stack:

• Drupal 7 ecosystem 
  
• Karaf (OSGI) under which coexist and interact series of features and bundles (camel, islandora-collection, islandora-image, islandora-sync, islandora-component, fcrepo4 camel)
  
• Fedora4 repository (4.4.0) + activeMQ broker
  
• blazeGraph(Big data) as triple store
  

Our workflow looks like this:

Drupal data definition:

     We use Drupal Nodes with rdf fields/image/etc -> bundles,  also know as content types, as our local representation of a fedora4 resource (not called object anymore)

     We have two types of content types right now: islandora collection and islandora basic image.

Ingest Workflow:

     When a user wants to ingest a new fedora4 resource (lets say an islandora basic image)

     It chooses a islandora basic image content type (node), fills out drupal node form with all the extra rdf fields (mappings) and submits it

     Drupal store this as a node inside Mysql (like it would for any normal drupal node derived content type)

     on submit we send a JSON representation (multipart) of this to our islandora-image endpoint which lives under Karaf, and is really a series of camel routing/processing/transformation chained processor. Think of this as a series of pipes.

     islandora-image gets from Fedora4 the info (RDF) for the parent resource (lets say a collection), completes the rdf with all the needed info and creates finally a Fedora4 resource using sparql + binary upload

     Fedora4 on it's side creates the resource and puts a message into the activeMQ broker

     Islandora-sync reads that message, logs into drupal and updates the already created node with the full info coming from Fedora4.

     We can put derivative creation/other post processing there. Means we don't have to wait for drupal anymore. We just need to make sure the preservation master is already in fedora.

     Note( there are a lot of intermediate steps missing, it's the big picture only)

So, the main idea is to make full use of Drupal Core and modules, and make every other process async. This architecture allows us further growing, scalability and a lot of goodies and also to reuse all other drupal contribute modules to make user interaction more dynamic. We are handling real drupal nodes.

But Drupal 7 is at the end, just a synchronous system, a very heavily developed one who is loved by all of you, but does not follow a modern event driven idea. This means we have do a lot of compromises and workarounds to make this work as we would love to. Drupal 8 on the other part, is a different beast. It's heavily based on a php development framework called Symphony 2, which includes a kernel that allows for event driven (routing also) system. 

What does this mean? That instead of creating a node and hooking like we do right now, there are events dispatched, which can be intercepted, routed, modified by other libraries/modules, etc.

This architecture is more close to what we (or I?) would love to have. More over, we can reuse a lot of external classes/components that are not written for drupal. Means our possibilities grow.

I have been doing some parallel work/thinking/planning on the drupal 8 side and islandora, moving from nodes to custom entities and looking for ways to move on this direction.. It's an personal effort, slow and lots of baby steps, which does not compromises the roadmap or any other work we are doing currently on D7, but my experience with this positive, so i would love to know what are/could/are not your feelings on this.

I don't think we have the resources to do both developments, D7 and D8. Also D8 is about to be released (new) but the development has being done but a lot of people for long time. The current RC4 is also very stable. Also, Daniel lamb and Nick have been doing an amazing job on Chullo, a component to talk directly to fedora4 that can be used directly on a D8 scenario.

Long post: I would love to have your feedback on this, so we can keep this discussion on multiple levels. I probably omitted a lot of stuff.

Thanks a lot

Diego Pino

[Nick Ruest]

Good stuff Diego! Thanks for getting this out on the list.

As it stands now, I'm -- personally -- not wedded to Drupal 7. If it
makes sense for us to pivot to Drupal 8, let's do it. But, that said, we
need a plan, and a some well articulated tickets/issues to work through IMO.

That said, this very topic is on the agenda[1] for today's CLAW Tech
Call. Hat tip to Nigel Banks for putting it on there :-)

So, I'd invite anybody in the community interested in this topic to join
us if you can. If not, I'm sure this will be on the next few weekly tech
call agendas.

cheers!

-nruest

[1] https://github.com/Islandora-CLAW/CLAW/wiki/November-18,-2015

On 15-11-18 11:55 AM, Diego Pino wrote:
> Dear Community,
>
> I wanted to share with you some ideas and also ask you for your opinion
> and feelings on the following topic:
>

> *Would it be a good idea to push efforts on our shinny new Islandora

> Claw project to develop for the upcoming (19 Nov) Drupal 8 instead of

> Drupal 7?*

>
> For those not involved yet in this new Islandora architecture, let me
> tell you what/how we are doing stuff right now (very simplified)
>
> Our current stack:
>

> * Drupal 7 ecosystem
> * Karaf (OSGI) under which coexist and interact series of features and

> bundles (camel, islandora-collection, islandora-image,
> islandora-sync, islandora-component, fcrepo4 camel)

> * Fedora4 repository (4.4.0) + activeMQ broker
> * blazeGraph(Big data) as triple store

>
> Our workflow looks like this:
>

> *Drupal data definition:*

> We use Drupal Nodes with rdf fields/image/etc -> bundles, also
> know as content types, as our local representation of a fedora4 resource
> (not called object anymore)
> We have two types of content types right now: islandora collection
> and islandora basic image.

> *Ingest Workflow:*

> When a user wants to ingest a new fedora4 resource (lets say an
> islandora basic image)
> It chooses a islandora basic image content type (node), fills out
> drupal node form with all the extra rdf fields (mappings) and submits it
> Drupal store this as a node inside Mysql (like it would for any
> normal drupal node derived content type)
> on submit we send a JSON representation (multipart) of this to our
> islandora-image endpoint which lives under Karaf, and is really a series
> of camel routing/processing/transformation chained processor. Think of
> this as a series of pipes.
> islandora-image gets from Fedora4 the info (RDF) for the parent
> resource (lets say a collection), completes the rdf with all the needed
> info and creates finally a Fedora4 resource using sparql + binary upload
> Fedora4 on it's side creates the resource and puts a message into
> the activeMQ broker
> Islandora-sync reads that message, logs into drupal and updates
> the already created node with the full info coming from Fedora4.
>
> We can put derivative creation/other post processing there. Means
> we don't have to wait for drupal anymore. We just need to make sure the
> preservation master is already in fedora.

> /Note( there are a lot of intermediate steps missing, it's the big
> picture only)/

> --
> For more information about using this group, please read our Listserv
> Guidelines: http://islandora.ca/content/welcome-islandora-listserv
> ---
> You received this message because you are subscribed to the Google
> Groups "islandora" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to islandora+...@googlegroups.com
> <mailto:islandora+...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/islandora.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/islandora/d19800d4-3ce0-4d00-8108-dc339e4eef57%40googlegroups.com
> <https://groups.google.com/d/msgid/islandora/d19800d4-3ce0-4d00-8108-dc339e4eef57%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Adon Irani]

Hello Diego, everyone

I'm new to the islandora world, but quite familiar with Drupal. I whole heartedly agree that the choice of supporting Drupal 8, at the expense of holding on to D7 support is prudent. It's a bit of a pay now or pay later scenario, or not putting new wine in the old wine sac as it goes.

One question which I don't have the immediate answer to: upgrade path. I've had OK results upgrading from D6 to D7. For a large production site on D7/islandora3, how much trouble would it be to migrate to D8 vs. rebuild from scratch? Difficulty on that front is still not enough, IMHO, to dissuade a D8 approach. But worth exploring.

For instance: https://www.drupal.org/upgrade/migrate suggests Views won't play nice yet for some structural reasons. That's a big issue potentially. But really, if it's a choice between having years of life on D8 and islandora4 at cost of rebuilding/rethinking some views vs. forego D8 to keep the status quo, I think that's an easy choice against the latter. Multi-lingual content and node references are also difficulties, which may be less trivial to work-around in the immediate sense. So will be informative to hear more from the community overall.

Just my quick thoughts...

Adon

[Diego Pino]

Thanks Adon for your thoughts. 

One of the things i don't fear is the migration process (only speaking for the Islandora side of things). In old Islandora we where basically using our own representation of what an "Object" was and using hooking abilities to do some nice stuff. Views was directly tied to our own Solr module implementation. Islandora CLAW will/should, in case of going this way, start from scratch, there is no migration really, the notion of object(resource right now) changed so much that moving old code is basically impossible. We will rethink, redo the needed functionality from the user perspective and also implement most of the previous custom assumptions the drupal way and leaving many of the Repository and Tripple transformations to external services.

But migration, for mixed Drupal sides can be an issue of course. I also agree with Nick that we need a plan, part of having a plan is knowing is community is willing to walk that road and their fears.

Thanks again for your comment.

Best

Diego

[Brad Spry]

Diego,

Ideas are now solidifying into requirements.

1. Events

With a solidified requirement, the question then becomes: Which Drupal version is better for events?

It appears D8 fulfills the requirement natively, while D7 must be retrofitted.   Which begs the question: will we be wasting precious resources trying to retrofit D7 with capabilities that are already ready to go in D8?

Digital repositories are powered by long-term thinking.  D8, with it's Symfony architecture, is clearly a product of long-term thinking and strategy.  I think it's a sea-change from Drupal's past; I honestly didn't think Drupal would have a long-term future, with Drie's no backwards compatibility mantra.  But with Symfony, Drupal has reinvented and positioned itself well for the future.

It seems like a choice between leveraging a ready-to-go toolbox (D8) or painstakingly reinventing and retrofitting each individual tool to make D7 D8-like or D8-lite.

I want the very best tools and support for Islandora developers, so they can make the most of their precious time and miraculous effort.  For that reason, D8, for all intents and purposes, appears to be the very best route for developers to take.

I believe the end result will be the best Islandora yet, most faithful to its vision, mission, and hopes, ultimately placing Islandora in the best position possible to survive the long haul.

Most Sincerely,

Brad Spry

UNC Charlotte Libraries

[Diego Pino]

Brad, 

Really thanks a lot for your thoughts, i agree on your vision and i fully support the long term/sustainability promise. It's also interesting to review how different our "mantras" at Islandora v/s Drupal 7 are. We try, whenever it's possible, to keep backwards compat. on base CMS system that does not care much about that!

Best

Diego Pino

[Donald Moses]

Thanks for posing the question Diego.  I think Brad captures many of the thoughts I had. 

Drupal 8 includes both REST and RDF in core - critical pieces of our developing Islandora/Fedora4 framework. Serialization is also part of D8 core and I think will be very important, but I don't know enough to express the why the moment. I think Symfony will be different, but longterm offers many benefits for developers and for the Islandora community - standards, reusable components, packages, a big development community, etc. 

Another Drupal 8 philosophy that I like is "proudly invented elsewhere" ... leveraging the best work done by the broader PHP community instead of reinventing the wheel.  My worry is that we will do the work in Drupal 7 (some of which may be unnecessary) and then we'll need to redo much of that work all over again. Given our limited community resources, we need to decide how best to focus those.

Donald

[Diego Pino]

Thanks to you Donald and an extra ++ for the "proudly invented elsewhere". I feel we all have very similar if not same requirements, ideas and wishes for our Islandora future and this also leads to making the best use possible of our resources avoiding duplicated efforts and reinventing wheels. I like to what this conversation is leading.

Best

Diego