Using multiple filesystems for archive

[Brendan]

Hello,

Is it possible to store data in a single XNAT archive across multiple file systems? I see that I can specify different locations for the pre-archive and archive and a few other things, but it seems the entire archive location is expected to be a single file system.  This seems to preclude using something like EXT4 for a large archive (tens of terabytes).

Thanks,
Brendan

[David Gutman]

So probably the easiest thing to do is do it at the file system level and have symlinks for specific archives.

for example my entire file system is in /home/xnat/xnat_data but that's an nfs mount to another point...

you should also be able to do something like /home/xnat/xnat_data/ARCHIVE/MY_PROJECT

and basically just mount your second/third/whatever filesystem at that path location.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
David A Gutman, M.D. Ph.D.
Assistant Professor of Biomedical Informatics
Senior Research Scientist, Center for Comprehensive Informatics
Emory University School of Medicine

[Ben Wagner]

Our IT department limits us to 300GB volumes.  We at first symlinked to new arc000n locations and change the appropriate section in the DB.  (See:  https://groups.google.com/d/topic/xnat_discussion/B2Q_yf4DlG0/discussion) There were odd problems with this approach.  Instead, we had to actually mount the drives at the arc000n location.  This has worked well.

Ben

[bennett landman]

This would be a great feature for us as well.

[David Gutman]

Looking back, I had similar issues with symlinking in the xnat_data/ARCHIVE directory, so what I wound up doing was just mounting additional storage directly within the xnat_data hierarchy when I needed additional space.

Assuming I had something like

/home/xnat/xnat_data/ARCHIVE/PROJECT_ONE and

/home/xnat/xnat_data/ARCHIVE/PROJECT_TWO

I did something similar to this (editing the /etc/fstab file)

gutmanraid.emory.edu:/XNAT_MORE_SPACE /MIGRATION_TEMP_MOUNT  nfs rw,intr,rsize=32768,wsize=32768,soft    1 2

So once the temporary mount point was set up, I'd rsync over the files from PROJECT_TWO to the XNAT_MORE_SPACE location, shutdown xnat, then unmount the /MIGRATION_TEMP_MOUNT and change the mount point to

/home/xnat/xnat_data/PROJECT_TWO

I'd then make sure I didn't fundamentally kill anything in PROJECT_TWO.... once that seemed kosher... there's still one trick/snafu.  I still have to delete the original "PROJECT_TWO" directory from the original file system otherwise I am not freeing up any space.  So I'd temporarily unmount the gutmanraid mount point,  since I've now verified and migrated over the files I can then delete the "original" copies on /home/xnat/xnat_data/ARCHIVE/PROJECT_TWO

However probably the safest way to do this is to just do this for new projects and not try and migrate/copy over old ones.  This also won't help with cases where a single project contains more than 300GB of data (For the example above).   I am not sure if setting up the system using LVM would get around that issue.....  I'm mostly a linux hack since I don't have to maintain production systems.

On Sat, Mar 2, 2013 at 11:02 PM, bennett landman <bala...@gmail.com> wrote:

This would be a great feature for us as well.

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Herrick, Rick]

Yeah, without playing stupid pet tricks with how you’re mounting file systems it’s not possible to have XNAT work across separate distinct file systems or servers. We’ve been looking at implementing the ability to integrate resources hosted in a variety of formats and locations (Zip archives, cloud-based storage, etc.) but not really any way to specify separate archive locations, e.g., on a project-by-project basis. That could be quite interesting, though…

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Jordi]

Hi all,

Let me re-take this old thread. 

We have a 20TB volume partition (eg. /data1) where one of our XNAT file systems currently resides, but it is slowly getting filled up. 

I could ask for a new identical volume (eg. /data2) mounted next to the main one, however what would be the best approach at this point for dealing with multiple XNAT archive volumes/filesystems? Is that still troublesome? Would it be preferable to extend its capacity and solely keep a main single one volume?

 

Regards,

Jordi

[Richard Cole]

Not a best practice, but upon "Project Creation", you could change the "Project Archive Folder" on partition 1 to a symbolic link pointing to the "Project Archive Folder" on partition 2.    I could see this getting pretty messy, especially if you have some Docker Swarm Workers who also need to mount the folders, but it's possible..

 

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/8d3a21d9-c750-47ba-861b-271f04439747n%40googlegroups.com.