Cannot upload .gz (non-tar) files

[Torsten Rohlfing]

Hi -

I think XNAT is making some inadequate inferences about uploaded
files, specifically, it assumes that any file with suffix ".gz" is
also a tar archive.

See, for instance, the following in
plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/
StoreXAR.java:

ZipI zipper = null;
if (compression_method.equalsIgnoreCase(".tar")){
zipper = new TarUtils();
}else if (compression_method.equalsIgnoreCase(".gz")){
zipper = new TarUtils();

zipper.setCompressionMethod(ZipOutputStream.DEFLATED);
}else{

Similar code is in (2x each)

plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/actions/
ExptFileUpload.java
plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/
ArcPut.java

perhaps more places.

As a result, our users cannot upload single files with gzip
compression.

I am wondering is that is really necessary?

Thanks!
Torsten

[Torsten Rohlfing]

While I am at it ... the not-quite-uncommon ".tgz" suffix, on the other hand, is notably absent from the list of detected suffixes :)

[Torsten Rohlfing]

Since we're mostly concerned about Tagged Uploads, the following two files may actually be more relevant:

plugin-resources/webapp/xnat/java/org/nrg/xnat/utils/CatalogUtils.java
plugin-resources/webapp/xnat/java/org/nrg/xnat/archive/UploadManager.java

Just a wild guess.

Thanks.

TR

[Rick Herrick]

All good points! I've added an issue to our issue tracker. I'm not sure when that will make it into the code base, but it's worth remembering, we're an open-source project and love contributions from the community ;)

[Torsten Rohlfing]

Thanks Rick:

In case this helps, here's how I would do it (with the caveat that I don't actually know Java at all), specifically in plugin-resources/webapp/xnat/java/org/nrg/xnat/utils/CatalogUtils.java:


               String compression_method = ".zip";
                if (filename.toUpperCase().endsWith( ".TAR" )) {
                        compression_method = ".tar";
                } else if ( filename.toUpperCase().endsWith( ".TGZ" ) || filename.toUpperCase().endsWith( ".TAR.GZ" ) ) {
                        compression_method = ".tgz";
                } else if (filename.indexOf(".") != -1) {
                        compression_method = filename.substring(filename.lastIndexOf("."));
                }
 

                if (extract && (compression_method.equalsIgnoreCase(".tar") || compression_method.equalsIgnoreCase(".tgz") || compression_method.equalsIgnoreCase(".zip") || compression_method.equalsIgnoreCase(".zar"))) {
                        File destinationDir = catFile.getParentFile();
                        final InputStream is = fi.getInputStream();

       
                        ZipI zipper = null;
                        if (compression_method.equalsIgnoreCase(".tar")) {
                                zipper = new TarUtils();

                        } else if (compression_method.equalsIgnoreCase(".tgz")) {

                                zipper = new TarUtils();
                                zipper.setCompressionMethod(ZipOutputStream.DEFLATED);
                        } else {

                                zipper = new ZipUtils();
                        }
       

Best,
  Torsten

[Torsten Rohlfing]

Okay - so further along these lines, here is a set of patches to XNAT 1.6.1 that we are currently test-driving.

These patches have the following effects:

- Files that are gzip-compressed but not tar archive (e.g., .nii.gz) can now be uploaded via the web interface without a problem

- Files in tar/gzip format with "combined suffix", i.e. ".tgz" instead of ".tar.gz" are now also properly treated as archives and their contents unpacked

Courtesy of NIAAA 1U01 AA021697 (N-CANDA Data Analysis) :)


Index: plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/actions/ExptFileUpload.java
===================================================================
--- plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/actions/ExptFileUpload.java  (revision 1789)
+++ plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/actions/ExptFileUpload.java  (working copy)
@@ -90,13 +90,17 @@
                     if(index>0)filename = filename.substring(index+1);
                    
                     String compression_method = ".zip";
-                    if (filename.indexOf(".")!=-1){
-                        compression_method = filename.substring(filename.lastIndexOf("."));
-                    }                  
-                   
+                    if (filename.toUpperCase().endsWith( ".TAR" )) {
+                            compression_method = ".tar";
+                    } else if ( filename.toUpperCase().endsWith( ".TGZ" ) || filename.toUpperCase().endsWith( ".TAR.GZ" ) ) {
+                            compression_method = ".tgz";
+                    } else if (filename.indexOf(".") != -1) {
+                            compression_method = filename.substring(filename.lastIndexOf("."));
+                    }
+                    
                     if (uploadID!=null)session.setAttribute(uploadID + "Upload", new Integer(100));
                     if (compression_method.equalsIgnoreCase(".tar") ||
-                            compression_method.equalsIgnoreCase(".gz") ||
+                            compression_method.equalsIgnoreCase(".tgz") ||

                             compression_method.equalsIgnoreCase(".zip") ||
                             compression_method.equalsIgnoreCase(".zar"))

                     {
@@ -106,7 +110,7 @@

                         ZipI zipper = null;
                         if (compression_method.equalsIgnoreCase(".tar")){
                             zipper = new TarUtils();

-                        }else if (compression_method.equalsIgnoreCase(".gz")){
+                        }else if (compression_method.equalsIgnoreCase(".tgz")){

                             zipper = new TarUtils();
                             zipper.setCompressionMethod(ZipOutputStream.DEFLATED);
                         }else{

Index: plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/ArcPut.java
===================================================================
--- plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/ArcPut.java  (revision 1789)
+++ plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/ArcPut.java  (working copy)
@@ -105,11 +105,15 @@
                     destination.mkdirs();
                 }
                 String compression_method = ".zip";
-                if (filename.indexOf(".")!=-1){
-                    compression_method = filename.substring(filename.lastIndexOf("."));
-                }                  
+                if (filename.toUpperCase().endsWith( ".TAR" )) {
+                        compression_method = ".tar";
+                } else if ( filename.toUpperCase().endsWith( ".TGZ" ) || filename.toUpperCase().endsWith( ".TAR.GZ" ) ) {
+                        compression_method = ".tgz";
+                } else if (filename.indexOf(".") != -1) {
+                        compression_method = filename.substring(filename.lastIndexOf("."));
+                }
                 if (compression_method.equalsIgnoreCase(".tar") ||
-                        compression_method.equalsIgnoreCase(".gz") ||
+                        compression_method.equalsIgnoreCase(".tgz") ||

                         compression_method.equalsIgnoreCase(".zip") ||
                         compression_method.equalsIgnoreCase(".zar"))

                 {
@@ -118,7 +122,7 @@

                     ZipI zipper = null;
                     if (compression_method.equalsIgnoreCase(".tar")){
                         zipper = new TarUtils();

-                    }else if (compression_method.equalsIgnoreCase(".gz")){
+                    }else if (compression_method.equalsIgnoreCase(".tgz")){

                         zipper = new TarUtils();
                         zipper.setCompressionMethod(ZipOutputStream.DEFLATED);
                     }else{

Index: plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/StoreXAR.java
===================================================================
--- plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/StoreXAR.java        (revision 1789)
+++ plugin-resources/webapp/xnat/java/org/nrg/xnat/turbine/modules/screens/StoreXAR.java        (working copy)
@@ -106,17 +106,20 @@
                 }
 
                 String compression_method = ".zip";
-                if (filename.indexOf(".")!=-1){
-                    compression_method = filename.substring(filename.lastIndexOf("."));
+                if (filename.toUpperCase().endsWith( ".TAR" )) {
+                        compression_method = ".tar";
+                } else if ( filename.toUpperCase().endsWith( ".TGZ" ) || filename.toUpperCase().endsWith( ".TAR.GZ" ) ) {
+                        compression_method = ".tgz";
+                } else if (filename.indexOf(".") != -1) {
+                        compression_method = filename.substring(filename.lastIndexOf("."));
                 }
-
                 InputStream is = fi.getInputStream();
                 System.out.println("Extracting file.");

 
                 ZipI zipper = null;
                 if (compression_method.equalsIgnoreCase(".tar")){
                     zipper = new TarUtils();

-                }else if (compression_method.equalsIgnoreCase(".gz")){
+                }else if (compression_method.equalsIgnoreCase(".tgz")){

                     zipper = new TarUtils();
                     zipper.setCompressionMethod(ZipOutputStream.DEFLATED);
                 }else{

Index: plugin-resources/webapp/xnat/java/org/nrg/xnat/utils/CatalogUtils.java
===================================================================
--- plugin-resources/webapp/xnat/java/org/nrg/xnat/utils/CatalogUtils.java      (revision 1789)
+++ plugin-resources/webapp/xnat/java/org/nrg/xnat/utils/CatalogUtils.java      (working copy)
@@ -460,11 +460,15 @@
                }
 
                String compression_method = ".zip";
-               if (filename.indexOf(".") != -1) {
-                       compression_method = filename.substring(filename.lastIndexOf("."));
-               }
+                if (filename.toUpperCase().endsWith( ".TAR" )) {
+                        compression_method = ".tar";
+                } else if ( filename.toUpperCase().endsWith( ".TGZ" ) || filename.toUpperCase().endsWith( ".TAR.GZ" ) ) {
+                        compression_method = ".tgz";
+                } else if (filename.indexOf(".") != -1) {
+                        compression_method = filename.substring(filename.lastIndexOf("."));
+                }
 
-               if (extract && (compression_method.equalsIgnoreCase(".tar") || compression_method.equalsIgnoreCase(".gz") || compression_method.equalsIgnoreCase(".zip") || compression_method.equalsIgnoreCase(".zar"))) {
+               if (extract && (compression_method.equalsIgnoreCase(".tar") || compression_method.equalsIgnoreCase(".tgz") || compression_method.equalsIgnoreCase(".zip") || compression_method.equalsIgnoreCase(".zar"))) {

[Torsten Rohlfing]

Just noticed that XNAT 1.6.3 still cannot upload gzip-compress files that aren't also tar archives... guess I'll have to patch my server again.

Wink, wink. :)

[Herrick, Rick]

Weird, I thought we'd fixed that. How are you trying to get the file into XNAT?

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/groups/opt_out.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Torsten Rohlfing]

Oh, I didn't try ... just checked the codebase.

CatalogUtils.java still says,

            ZipI zipper;

            if (compression_method.equalsIgnoreCase(".tar")) {
                zipper = new TarUtils();

            } else if (compression_method.equalsIgnoreCase(".gz")) {

                zipper = new TarUtils();
                zipper.setCompressionMethod(ZipOutputStream.DEFLATED);
            } else {

                zipper = new ZipUtils();
            }

which treats a .gz (non-tar) file as if it were a tar archive.

Admittedly, I did not check the other classes where this logic appeared, so some of them may actually be fixed.

[Herrick, Rick]

Right you are. I just created a bug in our issue tracker. We'll get that knocked out at some point.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

From: Torsten Rohlfing <torsten...@gmail.com>
Reply-To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Date: Thursday, February 20, 2014 6:28 PM
To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/groups/opt_out.

[Herrick, Rick]

Ah, it looks like it got fixed elsewhere when you upload an experiment to the archive. The CatalogUtils one would affect uploading resources to the archive, so it's a separate bit of code (our logic for determining compression methods should be componentized, but it's not; maybe that'll be something we'll tackle when we re-visit these issues).

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

[Torsten Rohlfing]

I will consider that "good news" since it means I didn't mouth off based on incorrect information and my own stupidity ;)

Thanks Rick!