irods and ceph object store: S3 or rados plugin?

[Len Zaifman]

We have an isilon front end and ceph archive system.

We are just building the ceph archive and plan on managing  the isilon data in place in irods and the data on the archive moved and managed by irods.

I see there is a newer S3 plugin interface requiring a compound resource with a filesystem based cache and a rados plugin which does not need the cache.

From the viewpoint of simplicity of implementation and operation, the rados plugin looks better- no cache to deal with no cleanup required when data is in ceph. However, we were hoping to start using irods 4.2.0 and the rados plugin was last updated 3 years ago from the git repository so I think it may not be compatible.

So it looks like the irods_resource_plugin_s3 is the way to go.

Are there any caveats to that S3 resource? Are there other options for rados integration into irods other than these 2 options?

A no cache required option would be nice.

Thank you.

[Jason Coposky]

The ceph plugin was never put into CI as we never had a production user.  If that is of interest, it is something we can certainly explore.  Let’s discuss this offline if you have interest in updating the ceph plugin for 4.2.

 

Thanks,

 

—
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675

m: (919)522-0517
jas...@renci.org
linkedin

twitter

irods.org

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Constable]

Having both 1 (and soon to be 4PB) of Ceph with a Rados Gateway in front of it and iRODS, we’d be interested in this as well as an additional option (especially given some hardware issues we’re having at the moment).

I confess I’d not realized this was ever available (especially cache-less). Not sure what we could offer in resources or time, but would be interested in the discussion.

 

John

-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

[Gijs Noorlander]

It surprises me the Rados plugin is not available anymore, since it seems such a promising feature.

Still it would be interesting to know the reason why it was discontinued (put on hold), or not requested by users.

[Terrell Russell]

Given this interest, we will dust off the plugin and get it up to speed for 4.2+.

  https://github.com/irods/irods_resource_plugin_rados

It won't be too much work to get the code ready, just need to work on getting it into CI so that is generally available and tested.

I'll be in contact with both of you about testing a new build after the User Group Meeting.

If anyone else is interested, please let me know.

Thanks!

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

[Andre]

Has anyone used this (ceph/irods with plugin) in production? 

Thanks,

Andre