Bitstream Architect - connect to multiple data stores

[Tianyi Gu]

Hello Tim, everyone,

I transformed the DSpace 7 Storage layer document into a visual diagram in the attachment.

Based on my understanding, DSpace only stores data to the one (if we have 10 stores) that has the biggest Store Number.

If I want DSpace to write data into several storages, each matching a designated community, sub-community or collection, on the same time, we have to set up the same numbers of DSpace instances on different cloud infrastructures? 

Tim, could you please give an exposure on some relative topics in the future update plan? It is crucial for my project plan as well.

Thank you,

Tianyi

[Tianyi Gu]

I updated the diagram

[Tim Donohue]

Hi Tianyi,

Currently, there are no plans on the Roadmap to change bitstream storage behavior in DSpace.  

The planned 7.x features can all be found at https://wiki.lyrasis.org/display/DSPACE/DSpace+Release+7.0+Status#DSpaceRelease7.0Status-Whatfeaturesarecominginalater7.xrelease?7.x

The Roadmap for 8.x or beyond are still under discussion by our DSpace Steering Group.

All that said, as an open-source project we always do accept donated contributions from community members.  So, if an institution developed (or hired a service provider to develop) a new feature or improvement, then it would be considered for a more immediate release (after undergoing a review by our Committers / Developers & also feedback from Steering and others as needed).  So, our Roadmap is never completely set in stone, but this sort of feature doesn't currently exist on our Roadmap.

Tim

---

From: dspac...@googlegroups.com <dspac...@googlegroups.com> on behalf of Tianyi Gu <gutia...@gmail.com>
Sent: Tuesday, October 26, 2021 7:17 AM
To: DSpace Technical Support <dspac...@googlegroups.com>
Subject: [dspace-tech] Bitstream Architect - connect to multiple data stores

 

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/406f64d1-0e8e-4a14-a1fc-bf73b5c26c05n%40googlegroups.com.

[Mark H. Wood]

On Tue, Oct 26, 2021 at 05:17:13AM -0700, Tianyi Gu wrote:
> I transformed the DSpace 7 Storage layer document into a visual diagram in
> the attachment.
>
> Based on my understanding, DSpace only stores data to the one (if we have
> 10 stores) that has the biggest Store Number.

No. See the file 'config/spring/api/bitstore.xml'. The
BitstreamStorageService is configured with a map of store numbers to
instances of BitStoreService implementations, and a property
'incoming' which selects from that map the store to which all new
bitstreams are deposited. To change the "incoming" store requires
editing this file and restarting DSpace.

> If I want DSpace to write data into several storages, each matching a
> designated community, sub-community or collection, on the same time, we
> have to set up the same numbers of DSpace instances on different cloud
> infrastructures?

I have given some thought to making the selection of assetstores a
pluggable policy, by abstracting BitstreamStorageService or by having
it refer to a configured implementation of a policy interface. My own
use cases would be to direct very large bitstreams to nearline storage
and smaller ones to local storage, or perhaps to producing backups by
"twinning" bitstreams to multiple stores when they are deposited. But
a policy implementation could be based on any computable criteria.

There is not yet any code to do this.

--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

[Tianyi Gu]

Hi Tim,

I will read the 7.x features. 

Personally, I like DSpace a lot. But I don't know how my stakeholders will compare DSpace with Fedora or Zenodo. 

Much appreciated!

Tianyi

[Tianyi Gu]

Hi Mark,

That's brilliant! 

I see what you mean. I can find it from 'config/spring/api/bitstore.xml':

https://github.com/DSpace/DSpace/blob/main/dspace/config/spring/api/bitstore.xml

To get the ideal stage, we need more development work!

I will let you know if I could do anything from my side.

Thank you,

Tianyi