dcm4chee ONLINE storage to S3

[jonm...@gmail.com]

A while ago I spent some time with Damien's code found here and was able to successfully get NEARLINE storage of tarballs of DICOM files to work correctly.  However, now I'm interested in bypassing storing DICOM files on the local disk at all and instead storing them directly on S3 via ONLINE storage (preferably without taring them).  Can someone tell me

1.  If this is already done (I doubt it)

2.  How to do this?

Ideally I'd like to know which parts already exist (ie, can I specify a different machine for my ONLINE storage, this would be a good starting point) and which parts I have to write myself.  My hunch is that I'll have to modify Damien's code and someone insert it into the ONLINE storage component, but I'm not sure how to do this.

Any help would be great.

Thanks

PS Right now I'm using version 2.17.0 and have that successfully deployed, so I'd only like to go through the upgrade pain if there are some specific features in a newer version which would help with this task.

[jonm...@gmail.com]

A few more details to help understand my predicament.

Right now I have successfully gotten dcm4chee deployed on EC2 using MySQL with AWS RDS, and everything works fine.  Ideally what I'd like to do is have all the DICOM files stored in S3.  This is good for me for 2 reasons

1.  S3 storage is more backed up than EBS.

2.  If there is no permanent data on the EBS volumes which are hooked up to the EC2 instances then I can have a load balancer as the forward facing DNS and any EC2 instance will work fine to process any request (GET or PUT of DICOM files) because all instances will talk to a shared DB (RDS) and all access from a shared storage location (S3).  This will not work properly if at any time DICOM files are stored on the individual EC2's EBS as the next request may get routed to a different EC2 instance who will ask the shared DB where to get the requested file.  The DB will respond with "from your local archive folder" which won't work because it's not on that particular EBS volume.  If the DB responded with "at this S3 location" then it would work across all EC2 instances.

I can provide more information if it would help.

Thanks

[Damien Evans]

I'm very familiar with your type of setup on AWS.  We have achieved load balancing with EBS by using GlusterFS to provide a NAS (shared network attached storage) for all of our app servers.  It works very well.  Gluster (http://www.gluster.org/) can aggregate multiple EBS volumes and cluster them to provide shared storage in AWS.

It would be quite a bit of work to use S3 for ONLINE storage in dcm4chee.  I've looked at it before and deemed it not to be worth the effort.  That being said, I do have a project coming up for work that will require me to dust off the S3 code that I wrote, update and re-test it to get it ready for production.  I'm curious about your experience with it.  How was your performance with S3?

 -- Damien

--
You received this message because you are subscribed to the Google Groups "dcm4che" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dcm4che+u...@googlegroups.com.
To post to this group, send email to dcm...@googlegroups.com.
Visit this group at http://groups.google.com/group/dcm4che?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[jonm...@gmail.com]

Thanks for your update, I will have a look at Gluster myself as well.  I've also seen S3FS, do you have any opinion between those?  Honestly I think I like the S3 storage more because then backups become almost unnecessary, as opposed to EBS based storage where you have to manage your own backups.  I may poke around at ONLINE storage a little bit with S3 to see what I can come up with.  As far as performance, I haven't done a lot of testing yet, as this is a relatively new project, nevertheless, the little performance testing I did do seemed to indicate that it was working well.  A few notes that I realized about your install procedure as well.

a.  replace all references to dcm4chee.archive:service=FileCopyHSMModule,type=S3 with dcm4chee.archive:service=FileCopyHSMModule,type=CLOUD

b.  the tar path referenced as tar:/storage/nearline, can be set to anything.  I have even set it to tar:S3.  As long as it is consistent it'll work.

In addition I added the following code to the end of the storeHSMFile method

        // This is added by JHM.  It would appear that the .tar files in the 

        // outgoing directory just sit around after they are uploaded.  If they

        // were successfully uploaded, then these just accumulate space, so 

        // let's go ahead and delete them here.

        fetchHSMFileFinished(fsID, filePath, file);

        return filePath;

If you have any more feedback, I'd welcome the information about either your GlusterFS setup or storage in S3.

Thanks

[Meditecs]

Hi Jonathan,

For an existing EC2 instance, I am trying to add a NEARLINE storage volume on S3 storage. As it looks you have succeeded in doing this, I was wondering if you could share your built S3 HSM module ?  I have tried to built it myself, but not being very familiar with Eclipse environment, I was hoping you might be able to help me out?

Kind regards

Hans

[Sachin Sharma]

Hello ,

i also installed dcm4chee version 2.18.1 with mysql version 5.7 on   aws ec2 server now i want to push data from machine like (CT, MR, Xray and mainly from OCT) to aws server now how will do that.

WE created node on machine like aetitle - DCM4CHEE , PORT NO.- 11112 & give IP - server ip so how can i insert data on dcm4chee because by this server ip and all that  i didn't get the data.

PLEASE SUGGEST HOW CAN I DO THAT. ANY PRIOR HELP NEEDED

[Rich Rosenbaum]

I've not tried it yet, and I'm not familiar with implications of using fuse, but I was thinking of just mounting an S3 bucket in the EC2 instance, and creating a symbolic link /mnt/S3bucket_path/archive > ...../server/default/archive. Why would that work, or not work?

Rich

[zahid saleem]

From: ijaz store, Pakistan. Dear businessman: You good self sells after all buying from somewhere? Can you buy from us? Do you want to check our prices? Whatsapp 00923150085681, zahidsal...@gmail.com, Apparel, CPU, Clothes, dry fruits, FMCG, food items, fruits, furniture, garments, groceries,  herbs, keyboards, laptops, mouse, monitors, perfumes, printers, sneakers, spare parts for Suzuki, Toyota, Honda cars, tablets, Toys for babies, kids, children for amusement, playing, education. etc. WHAT REWARD YOU PLEASE WANT. Registered office address: Ijaz store / corporation, Rawalpindi 46000, Pakistan, Whatsapp; 0092 315 0085681, Email; zahidsal...@gmail.com, National tax No. 7622921-2, Sales tax No. 3277876146819, (supplier, wholesellers, Government Contractors / builders /constructors) – PEC Licensed                                                                                                                                                                                                                

Warehouse address: SAME