new question about running FSL's bet via the container service, this time about how to get my output files back into XNAT

Katie Surrence

unread,

May 5, 2018, 9:30:19 PM5/5/18

to xnat_discussion

Dear all,

The last time I wrote I thought I had perhaps gotten my FSL/bet container working but I was sorely confused. However, after more reading and thinking I think I am substantially clearer now.

Here's where I am: in the latest version of my command, my container launches and runs without error, and I can generate a file. However, I am not saving the output file in the right place, and the files I have created are not being uploaded to XNAT.

Here is the mostly working version of the command:

{
  "name": "bet-derived",
  "label": "bet-derived",
  "description": "FSLBrainExtractionTool",
  "version": "1.0",
  "schema-version": "1.0",
  "info-url": "https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/BET/UserGuide",
  "image": "nipype/nipype:latest",
  "type": "docker",
  "index": "https://index.docker.io/v1/",
  "working-directory": null,
  "command-line": "bet /input/input_AX_FRFSE-XL_T2_20080822115111_3.nii output.nii.gz [OPTIONS]",
  "override-entrypoint": null,
  "mounts": [
    {
      "name": "nifti-in",
      "writable": false,
      "path": "/input"
    },
    {
      "name": "extracted-nifti-out",
      "writable": true,
      "path": "/output"
    }
  ],
  "environment-variables": {
    "envName1": "envVal1",
    "envName2": "#inputReplacementKey#"
  },
  "ports": {
    "22": "52222",
    "80": "8080"
  },
  "inputs": [
    {
      "name": "options",
      "description": "Command-line flags to pass to bet",
      "type": "string",
      "matcher": null,
      "default-value": null,
      "required": false,
      "replacement-key": "[OPTIONS]",
      "command-line-flag": null,
      "command-line-separator": null,
      "true-value": null,
      "false-value": null
    }
  ],
  "outputs": [
    {
      "name": "extracted-nifti",
      "description": "The nifti file with brain extracted from skull and CSF",
      "required": true,
      "mount": "extracted-nifti-out",
      "path": null,
      "glob": ""
    }
  ],
  "xnat": [
    {
      "name": "bet-nifti",
      "description": "Run FSL’s brain extraction on a nifti file.",
      "contexts": [
        "xnat:imageScanData"
      ],
      "external-inputs": [
        {
          "name": "scan",
          "description": "scan",
          "type": "Scan",
          "matcher": "'NIFTI' in @.resources[*].label",
          "default-value": null,
          "required": true,
          "replacement-key": null,
          "provides-value-for-command-input": "",
          "provides-files-for-command-mount": "",
          "via-setup-command": null,
          "user-settable": true,
          "load-children": true
        }
      ],
      "derived-inputs": [
        {
          "name": "scan-nifti",
          "description": "The nifti resource on the scan",
          "type": "Resource",
          "matcher": "@.label == 'NIFTI'",
          "default-value": null,
          "required": true,
          "replacement-key": null,
          "provides-value-for-command-input": "",
          "provides-files-for-command-mount": "nifti-in",
          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "scan",
          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        },
        {
          "name": "file-nifti",
          "description": "The nifti file on the resource",
          "type": "File",
          "matcher": "@.name == 'input_AX_FRFSE-XL_T2_20080822115111_3.nii'",
          "default-value": null,
          "required": true,
          "replacement-key": null,
          "provides-value-for-command-input": "",
          "provides-files-for-command-mount": "",
          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "scan-nifti",
          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        }
      ],
      "output-handlers": [
        {
          "name": "nifti-output-handler",
          "accepts-command-output": "extracted-nifti",
          "via-wrapup-command": null,
          "as-a-child-of-wrapper-input": "scan",
          "type": "Resource",
          "label": "NIFTI"
        }
      ]
    }
  ],
  "reserve-memory": null,
  "limit-memory": null,
  "limit-cpu": null
}

After this runs, I do a find for 'output.nii.gz' and I discover that the file exists in a directory called:









/var/lib/docker/overlay2/<long container id>/diff/work

That diff directory also has subdirectories input and output as siblings to work, and I suspect I wanted my file to go into output. But, how? I tried changing `command-line` to `"bet "/input/input_AX_FRFSE-XL_T2_20080822115111_3.nii /output/output.nii.gz [OPTIONS]"

But that just gets me the following in stderr when I try to run my container:









Error: failed to open file /output/output.nii.gz
Image Exception : #22 :: ERROR: Could not open image /output/output
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
Aborted (core dumped)

What I think is the analog on my local machine, making a directory and giving the output path as the path to a not yet existent file within that directory, works fine.

Can anyone tell me out I get my output file to be a new resource on the scan?

Thanks in advance!

Best,

Katie

John Flavin

unread,

May 7, 2018, 9:08:03 AM5/7/18

to XNAT Discussion board

The second version—where you write the file to /output—is correct. You should be writing all files that you want uploaded into your output mount(s). When your container finishes successfully, the container service looks through all the container's output handlers, finds the outputs they handle, and uploads the files in those outputs to the place you specified.

As to why BET is failing when you give it an output file in /output, I'm not sure. Could it be that your input file is .nii but your output file is .nii.gz? I will try to run a test later to reproduce the error you’re seeing.

If you can't figure it out, I recommend asking the FSL folks on their mailing list: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=FSL. Or just search that list with some of the error strings you have seen in case anyone else has had the same problem.

Good luck! You've done great work so far. You’re pretty close to getting this command to work. Once you’re over this last error, I think it will happen. Then the last thing I can see to fix is to not hard-code the name of the input file, so you can use this on other scans with different file names.

Flavin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

Katie Surrence

unread,

May 7, 2018, 12:38:24 PM5/7/18

to xnat_di...@googlegroups.com

Hi Flavin,

Thanks again for your quick, helpful responses. I think the source of this problem may in fact not be XNAT, or my understanding of it, but I thought I'd report a weird clue I have in case anyone on this list has seen this problem before.

Googling the error shows that it's basically what happens when you try to open a file that doesn't exist. Of course, the output file is not supposed to exist. But this error *would* make sense if the path where I want to write the output file is not writeable. And that actually makes some sense to me, because when I fire up my XNAT vm, and from there run my nipype:nipype docker container with an interactive shell, the root path / is indeed not writeable. I can't create a directory there. I can create a directory in the directory /work. I verified in my interactive container session that `bet /work/input/<theinputfile.nii> /work/output/output.nii.gz` worked fine. (And I've tried it with lots of different output file names. They all *should* work, because they all do work anywhere else other than in the docker container launched by XNAT.) So I tried substituting /work/input and /work/output for my mount paths, and changing the command to reflect that. That fails with the same error.

```

Error: failed to open file /work/output/output.nii.gzImage Exception : #22 :: ERROR: Could not open image /work/output/outputterminate called after throwing an instance of 'RBD_COMMON::BaseException'Aborted (core dumped)```

The weird clue I have is this. I'm now calling the label on my output handler `NIFTI-EXTRACTED`. When I run the command with just output.nii.gz, now on the 4th scan in this session, and I go to /data/xnat/archive/XNAT_2016/arc001/case057_MR1/SCANS/4/NIFTI-EXTRACTED, I see a single XML file, 4_catalog.xml. When I try to run it with either my /output or my /work/output paths, I see the following in that directory:

4_catalog.xml bzip2 echo kmod mt ntfswipe ss umount

bash bzip2recover ed less mt-gnu open static-sh uname

btrfs bzless egrep lessecho mv openvt stty uncompress

btrfs-calc-size bzmore false lessfile nano pidof su unicode_start

btrfsck cat fgconsole lesskey nc ping sync vdir

btrfs-convert chacl fgrep lesspipe nc.openbsd ping6 systemctl wdctl

btrfs-debug-tree chgrp findmnt ln netcat plymouth systemd which

btrfs-find-root chmod fsck.btrfs loadkeys netstat ps systemd-ask-password whiptail

btrfs-image chown fuser login networkctl pwd systemd-escape ypdomainname

btrfs-map-logical chvt fusermount loginctl nisdomainname rbash systemd-hwdb zcat

btrfs-select-super cp getfacl lowntfs-3g ntfs-3g readlink systemd-inhibit zcmp

btrfs-show-super cpio grep ls ntfs-3g.probe red systemd-machine-id-setup zdiff

btrfstune cpp gunzip lsb ntfs-3g.secaudit rm systemd-notify zegrep

btrfs-zero-log crda gzexe lsblk ntfs-3g.usermap rmdir systemd-tmpfiles zfgrep

bunzip2 dash gzip lsmod ntfscat rnano systemd-tty-ask-password-agent zforce

busybox date hostname mkdir ntfscluster run-parts tailf zgrep

bzcat dd ifupdown mkfs.btrfs ntfscmp sed tar zless

bzcmp df initrd.img.old mknod ntfsfallocate setfacl tempfile zmore

bzdiff dir ip mktemp ntfsfix setfont touch znew

bzegrep dmesg journalctl modprobe.d ntfsinfo setupcon true

bzexe dnsdomainname kbd_mode more ntfsls sh udevadm

bzfgrep domainname keyctl mount ntfsmove sh.distrib ufw

bzgrep dumpkeys kill mountpoint ntfstruncate sleep ulockmgr_server

Trying to cd around the docker container in interactive mode, some of those files I recognize as coming from /bin. But not all of them. I can't figure out where all of them come from and I apparently don't have sufficient permissions in the container to `find`. And I certainly don't know why some weird mix of files and directory is being understood by XNAT to be the output of my command.

I guess my next debugging step would be to try to make my own docker container/use a different one to try to see if I can get this to run. I chose nipype because I want to move on to making nipype executable with the container service. But maybe I can make my own nipype container.

Thanks again for all your help, and if you or anyone else has any insight triggered by this further information I'd appreciate it!

Flavin

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsub...@googlegroups.com.
To post to this group, send email to xnat_discussion@googlegroups.com.

Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@googlegroups.com.
To post to this group, send email to xnat_discussion@googlegroups.com.

Katie Surrence

unread,

May 7, 2018, 3:54:02 PM5/7/18

to xnat_discussion

an update/some progress:

I was right that a part of the problem was the image/container (probably something about its write permissions). I have switched to using brainlife/fsl and I no longer get that error when I try to run the command with /input/<the input .nii > and /output/<an output file name>.

When I run my same command with containers derived from this new image, the output.nii.gz file is now winding up in a directory called /data/xnat/build/<long identifier that belongs to I'm not sure what>/

That all seems like progress. However, I thought I would expect to see my files available under scans/4/resources/NIFTI-EXTRACTED, and I don't.

Also, just as with the other image, the directory

/data/xnat/archive/XNAT_2016/arc001/case057_MR1/SCANS/4/NIFTI-EXTRACTED

is getting written with: