nearline storage configuration help
Ajay
Ajay
Ajay
FileSystem[pk=1, tar:/storage/nearline, groupID=NEARLINE_STORAGE, aet=DCM4CHEE, NEARLINE, RW+, userinfo=null] FileSystem[pk=2, archive, groupID=ONLINE_STORAGE, aet=DCM4CHEE, ONLINE, RW+, userinfo=null]
Is there any wrong with this? The files are not getting deleted with the above configuration?
Does dcm4chee consider available storage space on online inorder to delete the study? If so, how can we skip this check?
Ajay
Aaron Boxer
--
You received this message because you are subscribed to the Google Groups "dcm4che" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dcm4che+u...@googlegroups.com.
To post to this group, send email to dcm...@googlegroups.com.
Visit this group at http://groups.google.com/group/dcm4che.
For more options, visit https://groups.google.com/groups/opt_out.
Ajay
leogrande
From that article: "...
Version InfoFYI - This feature is only available by building the code from SVN right now. http://dcm4che.svn.sourceforge.net/viewvc/dcm4che/dcm4chee/sandbox/dcm4chee-hsm-cloud/.."
Do you have that Amazon S3 HSM Module in the list of services in the JMX console?
Andres Castiblanco
Thanks in advance.
matthe...@netscape.net
Use version 2.18.3
Don't need to build anything
You are on windows? Found this on a google search: http://forums.dcm4che.org/jiveforums/thread.jspa?threadID=3401
In this section we will configure the various services and get an understanding of what they do.
-
Add a file system to the NEARLINE storage group
-
Open a web browser, and navigate to the JMX console. e.g. http://localhost:8080/jmx-console
-
Locate the “dcm4chee.archive” section, and click on “group=NEARLINE_STORAGE,service=FileSystemMgt”
-
Scroll down to the “List of MBean Operations” section and find the “addRWFileSystem()” operation. Enter in a path for nearline storage. This will not actually be used, since we are going to store the files in S3, but we need to configure something here so that the system knows we are using the NEARLINE storage. I have entered: “tar:/storage/nearline”. Note the tar prefix. This tells dcm4chee that all of the files going to this storage group will be tarred up.
-
Click Invoke to create add the file system record into the database.
-
-
Configure the FileCopy service
The FileCopy service is responsible for physically copying files to your nearline storage. This is where we configure our particular plugin.-
Specify a value for the DestinationFileSystem. This value should equal the value you specified for your nearline storage file system so that dcm4chee knows that this FileCopy service is associated with that file system configuration. e.g. “tar:/storage/nearline”
-
Specify a value for HSMModulServicename. This should be the JMX ObjectName of our S3 plugin module, and enables it for use within this service when storing and retrieving files. Enter: “dcm4chee.archive:service=FileCopyHSMModule,type=S3”
-
Leave the FileStatus set to TO_ARCHIVE. This will be the status of files stored in S3. When the SyncFileStatus service runs and verifies that these files are stored properly, it will change the status to ARCHIVED.
-
Click Apply Changes.
-
-
Configure the TarRetriever service
This service is responsible for fetching and extracting tar files from the nearline storage during retrieve requests.-
Specify a value for HSMModulServicename. This should be the JMX ObjectName of our S3 plugin module, and enables it for use within this service when storing and retrieving files. Enter: “dcm4chee.archive:service=FileCopyHSMModule,type=S3”
-
Click Apply Changes.
-
-
Configure the S3 HSMModule (service=FileCopyHSMModule,type=S3)
Now we are ready to configure the S3 integration. Here are the main things to configure here:-
Amazon S3 bucket name
-
Amazon AWS Access Key
-
Amazon AWS Secret Key (this is write only, and you will not see a value after clicking Apply Changes)
-
The Outgoing and Incoming directories in this configuration are temporary storage areas that are used for tarring and untarring files.
-
Click Apply Changes.
-
-
Configure the SyncFileStatus service
This service will run periodically and verify the files that have been stored to S3. It will fetch the tar files and ensure that the correct files are contained within. Once it verifies the files, it will update the file status in the database to ARCHIVED.-
Specify a value for the MonitoredFileSystem. This value should equal the value you specified for your nearline storage file system so that dcm4chee knows that this service is associated with that file system configuration. e.g. “tar:/storage/nearline”
-
Specify a value for HSMModulServicename. This should be the JMX ObjectName of our S3 plugin module, and enables it for use within this service when fetching tar files. Enter: “dcm4chee.archive:service=FileCopyHSMModule,type=S3”
-
Specify a TaskInterval. It is set to NEVER by default, so you you should set it to a proper interval that is good for your workflow, preferably not during peak business hours.
-
Summary
At this point you should be able to store DICOM objects to dcm4chee and it will archive them to Amazon S3. The S3 key will be a hierarchical path, which should look familiar to you if you have looked at how dcm4chee stores objects on a file system. For example, here is a screenshot of my Amazon Management Console showing the archived path:
In the database, the files should have a changed file status, and should reflect their tar path as shown in this screenshot:
If you can’t read it in the picture, the filepaths look like this: 2011/8/3/16/745ABFED/CF024730-323397.tar!CF024730/000004A0
Note the “tar” designator in the path. This tells the system that the file is contained within a tar file.
Setting up Retention Rules to Remove Studies from Online
We probably don’t want two copies of the study forever, so lets set up some rules now so that studies are deleted from the ONLINE storage group after a period of time. This will leave the remaining copy on S3.
Note that this is only an example. Your retention/deletion requirements may differ!
-
Configure deletion of ONLINE studies
-
Open a web browser, and navigate to the JMX console. e.g. http://localhost:8080/jmx-console
-
Locate the “dcm4chee.archive” section, and click on “group=ONLINE_STORAGE,service=FileSystemMgt”
-
Set DeleteStudyIfNotAccessedFor = your retention period (52w or whatever your SLA requires)
-
Set DeleteStudyOnlyIfStorageNotCommited = false
-
Set DeleteStudyOnlyIfCopyOnMedia = false
-
Set DeleteStudyOnlyIfCopyOnReadOnlyFileSystem = false
-
Set ScheduleStudiesForDeletionInterval = a reasonable time interval for the system to check the database and schedule deletion jobs.
-
Set DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup = NEARLINE_STORAGE
-
Set DeleteStudyOnlyIfCopyArchived = true (only delete studies that have been verified by the SyncFileStatus service. If you don’t care about that or are not running that service, you can set this false.)
-
Click Apply Changes
-
At this point, dcm4chee will look for studies in ONLINE that meet these criteria and schedule them for deletion. After they are deleted, and the only copy is on S3, a retrieve request will trigger a fetch from Amazon. The tar file(s) will be fetched, images extracted and sent to the destination.
That’s it!
-
Added by Damien Evans, last edited by Jan Pechanec on May 04, 2011 (view change)
![Alvaro G. [Andor]'s profile photo](http://lh3.googleusercontent.com/a-/ALV-UjV4h-pCBR-qChQH0OnM_N1elA3ONFoHdgaBVvfGz2af4qEehxvR=s40-c)
Alvaro G. [Andor]
- Install Castor, which AFAIK it's not called "Castor" any more, and I think a license and an additional server is needed for that, isn't it?
- Configure an external storage module for Castor that connects to S3
- Connect dcm4chee to Castor
Use version 2.18.3
No need to build the code.
In this section we will configure the various services and get an understanding of what they do.
1. Add a file system to the NEARLINE storage group1. Open a web browser, and navigate to the JMX console. e.g. http://localhost:8080/jmx-console2. Locate the “dcm4chee.archive” section, and click on “group=NEARLINE_STORAGE,service=FileSystemMgt”3. Scroll down to the “List of MBean Operations” section and find the “addRWFileSystem()” operation. Enter in a path for nearline storage. This will not actually be used, since we are going to store the files in S3, but we need to configure something here so that the system knows we are using the NEARLINE storage. I have entered: “tar:/storage/nearline”. Note the tar prefix. This tells dcm4chee that all of the files going to this storage group will be tarred up.4. Click Invoke to create add the file system record into the database.2. Configure the FileCopy service
The FileCopy service is responsible for physically copying files to your nearline storage. This is where we configure our particular plugin.
1. Specify a value for the DestinationFileSystem. This value should equal the value you specified for your nearline storage file system so that dcm4chee knows that this FileCopy service is associated with that file system configuration. e.g. “tar:/storage/nearline”2. Specify a value for HSMModulServicename. This should be the JMX ObjectName of our S3 plugin module, and enables it for use within this service when storing and retrieving files. Enter: “dcm4chee.archive:service=FileCopyHSMModule,type=CAStor ”3. Leave the FileStatus set to TO_ARCHIVE. This will be the status of files stored in S3. When the SyncFileStatus service runs and verifies that these files are stored properly, it will change the status to ARCHIVED.4. Click Apply Changes.3. Configure the TarRetriever service
This service is responsible for fetching and extracting tar files from the nearline storage during retrieve requests.
1. Specify a value for HSMModulServicename. This should be the JMX ObjectName of our S3 plugin module, and enables it for use within this service when storing and retrieving files. Enter: “dcm4chee.archive:service=FileCopyHSMModule,type=CAStor”2. Click Apply Changes.4. Configure the CAStor HSMModule (service=FileCopyHSMModule,type=CAStor)
Here are the main things to configure here:
1. IP address of storage node2. How long to store image (in weeks, 52000 is 100 years)
5. Configure the SyncFileStatus service
This service will run periodically and verify the files that have been stored to S3. It will fetch the tar files and ensure that the correct files are contained within. Once it verifies the files, it will update the file status in the database to ARCHIVED.
1. Specify a value for the MonitoredFileSystem. This value should equal the value you specified for your nearline storage file system so that dcm4chee knows that this service is associated with that file system configuration. e.g. “tar:/storage/nearline”2. Specify a value for HSMModulServicename. This should be the JMX ObjectName of our CAStor plugin module, and enables it for use within this service when fetching tar files. Enter: “dcm4chee.archive:service=FileCopyHSMModule,type=CAStor”3. Specify a TaskInterval. It is set to NEVER by default, so you you should set it to a proper interval that is good for your workflow, preferably not during peak business hours.
SummaryAt this point you should be able to store DICOM objects to dcm4chee and it will archive them to Amazon S3. The S3 key will be a hierarchical path, which should look familiar to you if you have looked at how dcm4chee stores objects on a file system. For example, here is a screenshot of my Amazon Management Console showing the archived path:
In the database, the files should have a changed file status, and should reflect their tar path as shown in this screenshot:
If you can’t read it in the picture, the filepaths look like this: 2011/8/3/16/745ABFED/CF024730-323397.tar!CF024730/000004A0Note the “tar” designator in the path. This tells the system that the file is contained within a tar file.Setting up Retention Rules to Remove Studies from Online
We probably don’t want two copies of the study forever, so lets set up some rules now so that studies are deleted from the ONLINE storage group after a period of time. This will leave the remaining copy on CAStor.
Note that this is only an example. Your retention/deletion requirements may differ!
1. Configure deletion of ONLINE studies1. Open a web browser, and navigate to the JMX console. e.g. http://localhost:8080/jmx-console2. Locate the “dcm4chee.archive” section, and click on “group=ONLINE_STORAGE,service=FileSystemMgt”3. Set DeleteStudyIfNotAccessedFor = your retention period (52w or whatever your SLA requires)4. Set DeleteStudyOnlyIfStorageNotCommited = false5. Set DeleteStudyOnlyIfCopyOnMedia = false6. Set DeleteStudyOnlyIfCopyOnReadOnlyFileSystem = false7. Set ScheduleStudiesForDeletionInterval = a reasonable time interval for the system to check the database and schedule deletion jobs.8. Set DeleteStudyOnlyIfCopyOnFileSystemOfFileSystemGroup = NEARLINE_STORAGE9. Set DeleteStudyOnlyIfCopyArchived = true (only delete studies that have been verified by the SyncFileStatus service. If you don’t care about that or are not running that service, you can set this false.)10. Click Apply Changes
At this point, dcm4chee will look for studies in ONLINE that meet these criteria and schedule them for deletion. After they are deleted, and the only copy is on S3, a retrieve request will trigger a fetch from Amazon. The tar file(s) will be fetched, images extracted and sent to the destination.That’s it!
Added by Damien Evans, last edited by Jan Pechanec on May 04, 2011 (view change)
On Friday, February 12, 2016 at 7:04:22 AM UTC-8, Andres Castiblanco wrote:
--
You received this message because you are subscribed to the Google Groups "dcm4che" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dcm4che+u...@googlegroups.com.
To post to this group, send email to dcm...@googlegroups.com.
Visit this group at https://groups.google.com/group/dcm4che.
For more options, visit https://groups.google.com/d/optout.
