Call for participation: F4 Asynchronous Storage
Cowan, William G
Hello All,
As you may know, Andrew Woods started a conversation last May [0] soliciting feedback for post-4.2.0 Fedora feature development. From that, it was determined that the top 3 potential features for community development should be focused on WebAccessControl, Asynchronous Storage, and Transparent Filesystems. Work in the past month has already produced a positive start towards WebAccessControl, and the development teams at DuraSpace, Indiana University, and Amherst College would like to bring Asynchronous Storage into focus, starting with a broader call for participation.
This feature is envisioned to focus on two different aspects related to storage interactions. On the back-end it would allow Fedora to communicate asynchronously with high-latency systems, including hierarchical storage management systems and cloud storage. On the front-end the Fedora REST-API would be updated to expose client interactions to help mediate asynchronous access to high-latency storage systems.
Actions taken to date include a rough sketch design by David Wilcox and Andrew Woods [1], and a recent conference call including the DuraSpace team, Indiana University, and Amherst College [2]. Given that the topics of pluggable and/or high-latency storage types have been discussed in various forms over a longer time period than the recent calls for feedback, we believe that interest in this area of development is potentially broader than the current representation. Given that its applicability is probably fairly high, the implementation probably demands careful guidance from the Fedora community from a design perspective, and hopefully others will wish to contribute development resources towards the effort as well.
Specific actions from the initial conference call include initiating this call for participation, and to begin thinking about whether or not the design should include the use of the API Extension Architecture work also currently in progress. Next steps would be a simple response from the community to this thread to help gauge interest, followed by additional conference calls as needed. While this message has been posted to multiple lists, please direct your responses to fedora-c...@googlegroups.com.
Regards,
[0] https://groups.google.com/d/topic/fedora-tech/9IrRXC2k5ZI/discussion
[1] https://wiki.duraspace.org/display/FF/Design+-+Asynchronous+and+Pluggable+Storage
[2] https://wiki.duraspace.org/display/FF/2015-08-17+-+Indiana+-+Amherst+F4+Storage
Brad Spry
I have a production Fedora 3.5 system that uses S3 for datastreamStore.
Fedora's objectStore's write operations were too intensive for S3, so I had to use SSD for objectStore and resourceIndex.
SSD costs three times more than S3, so I'm very interested in the possibility of Fedora being more asynchronous storage friendly!
I use YAS3FS to mount S3, it also has a robust local cache method with messaging capability, which can keep multiple servers in sync by sending a signal to the other servers to invalidate their local caches in anticipation of newly arrived files. You can also mount S3 read-only, I do for our public-facing server.
https://github.com/danilop/yas3fs
Another great asynchronous storage utility I use is MT-AWS-GLACIER for Glacier file management. I use it to send BagIt bags into Glacier for long haul storage.
https://github.com/vsespb/mt-aws-glacier
I only mention YAS3FS and MT-AWS-GLACIER as good examples, maybe there is something you can learn from them, I certainly did.
Sincerely,
Brad Spry
Atkins Library
UNC Charlotte