/opt/juicer/scripts/splits/*_R1*.fastq: No such file or directory

[Heather Wick]

Hi,

I'm attempting to run juicer locally on my mac but it won't get past the splits step (see below). I'm not sure if this is a bug or something that I did wrong. I downloaded all of the files in the CPU folder on github, but I noticed that that folder did not contain all of the files listed in the scripts folder of the "directory structure" section on the Aiden lab webpage regarding juicer usage.

Is there a more comprehensive guide for installation and use of juicer on a local machine, or could someone perhaps give a little more direct guidance in response to this? It's unclear whether I need files additional to the ones in the CPU folder on github. There is a folder called "common" in the CPU section that has some of the files mentioned in the scripts folder in the "directory section" on the Aiden lab page, but I"m not sure if I"m supposed to move them into scripts, or leave them in the "common" folder. Also, not all the files listed in the "directory structure" section are available in the "common" folder -- notably, the splits_rmdups.awk . Could either issue be why juicer won't get past the splits stage? Or does running it locally not require certain files or require a different directory structure?

Here is how I am running juicer.sh. I did not use -d because I actually just change the hardwired top directory directly in the script. Note that the splits folder is empty after running it, even though the fastq folder is not

bash juicer.sh -y /Path/to/opt/juicer/restriction_sites/HindIII_site_hg19.fixed.bed

(-: Looking for fastq files...fastq files exist

Mon Jan  9 11:54:04 EST 2017

Juicer version:1.5

juicer.sh -y /Path/to/opt/juicer/restriction_sites/HindIII_site_hg19.fixed.bed

(-: Aligning files matching /Path/to/opt/juicer/scripts/fastq/*_R*.fastq*

 in queue  to genome hg19 with site file /Path/to/opt/juicer/restriction_sites/HindIII_site_hg19.fixed.bed

---  Using already created files in /Path/to/opt/juicer/scripts/splits

paste: /Path/to/opt/juicer/scripts/splits/*_R1*.fastq: No such file or directory

ls -l splits/

ls -l fastq/

total 95566912

 24465125667 Jan  6 12:30 001_Control_R1.fastq

 24465125667 Jan  6 13:56 001_Control_R2.fastq

And here is what is in my script folder, common folder, which I had downloaded from the CPU page on github

pwd

/Path/to/opt/juicer/scripts

ls -l

total 66560

        68 Jan  6 16:04 HIC_tmp

       195 Jan  6 14:29 README.md

        68 Jan  9 11:54 aligned

       544 Jan  6 14:29 common

       136 Jan  6 16:13 fastq

  34047466 Jan  6 14:23 juicebox_tools.7.0.jar

     22345 Jan  6 15:31 juicer.sh

        68 Jan  6 16:15 splits

ls -l common/

total 66696

      4659 Jan  6 14:29 LibraryComplexity.class

      7204 Jan  6 14:29 LibraryComplexity.java

      4237 Jan  6 14:29 check.sh

     15345 Jan  6 14:29 chimeric_blacklist.awk

      1708 Jan  6 14:29 cleanup.sh

      3584 Jan  6 14:29 collisions.awk

      1616 Jan  6 14:29 countligations.sh

      5326 Jan  6 14:29 dups.awk

      3551 Jan  6 14:29 fragment.pl

      2031 Jan  6 14:29 juicebox

  34050850 Jan  6 14:29 juicebox_tools.7.0.jar

      3576 Jan  6 14:29 juicer_postprocessing.sh

     14513 Jan  6 14:29 statistics.pl

      1751 Jan  6 14:29 stats_sub.awk

Any assistance would be greatly appreciated.

Thank you,

~ Heather

[Neva Durand]

Hello Heather,

I’m sorry about the documentation, we’re working to make it more clear. Also one disclaimer: Juicer is designed to be run on a cluster system, not on a single computer, so apologies for the bugs, it’s still in beta.

You say you changed the hardwired topDir. Did you make sure it was an absolute path?

From other people’s problems, I think this is likely a problem with the soft linking.

Generally I would keep your scripts directory separate. Here is what I suggest:

Follow the same structure is as in the Juicer AWS mirror. You can see it here: https://bcm.box.com/v/juicerawsmirror
You can also download files from there to make things easier.

Make a folder “work” and put your fastqs there:

cd /Path/to/opt/juicer
mkdir work
cd work
mv /Path/to/opt/juicer/scripts/fastq .

Then you should go ahead and remove the directories created in your original folder (and you have an extra jar for some reason that should be removed):

rm -r /Path/to/opt/juicer/scripts/HIC_tmp

rm -r /Path/to/opt/juicer/scripts/aligned

rm /Path/to/opt/juicer/scripts/juicebox_tools.7.0.jar

Then the structure is as in the Juicer AWS mirror. You can see it here: https://bcm.box.com/v/juicerawsmirror
You can also download files from there to make things easier.

You should have your references and restriction sites under the same directory as scripts:

mkdir /Path/to/opt/juicer/references
mkdir /Path/to/opt/juicer/restriction_sites

Put your fasta file and the bwa index files in references and your restriction site file in restriction_sites.
We have these in the Box link above. Let’s suppose you were doing human with the MboI restriction site. Download Homo_sapiens_assembly19.fasta and all the other files with Homo_sapiens_assembly19 as the prefix from the references folder on Box. Then download the hg19_MboI.txt file from the restriction_sites folder.

cd /Path/to/opt/juicer/
ls references
ls restriction_sites

This should show 6 files starting with Homo_sapiens_assembly19 in references and hg19_MboI.txt in restriction_sites

Then go back to your work folder and run juicer, telling it where the scripts are:

cd /Path/to/opt/juicer/work
/Path/to/opt/juicer/scripts/juicer.sh -D /Path/to/opt/juicer/ -g hg19 -s MboI

Let me know if that works.

Best
Neva

​

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/557d579b-ead6-4afe-84ef-2a9769400d9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Neva Cherniavsky Durand, Ph.D.

Staff Scientist, Aiden Lab

www.aidenlab.org

[Heather Wick]

Thanks for the speedy response!
I was confused about the fastq being moved to the work folder, as on the Aiden lab site and in the directory you sent it gives an example such as:

/Users/hwick/opt/juicer/work/MBR9/fastq/XXX.fastq.gz

As it turns out though, juicer complains if the fastq folder is not directly under work/ (ie, it doesn't tolerate fastq being in the MBR9 folder).

I downloaded all the scripts from the directory system you sent to replace my set of scripts. I had some wonky issues with it claiming my HindIII file didn't exist and needed to be created; so I downloaded the one from the directory. Got the same error, tried changing the name to remove "_new", ran it again, somehow juicer then deleted the file, so I redownloaded it, ran it, and got this instead:
 Heathers-MacBook-Pro-3:work hwick$ /Users/hwick/opt/juicer/scripts/juicer.sh -D /Users/hwick/opt/juicer -g hg19 -s HindIII

(-: Looking for fastq files...fastq files exist

/Users/hwick/opt/juicer/scripts/juicer.sh: line 347: bsub: command not found
(-: Aligning files matching /Users/hwick/opt/juicer/work/fastq/*_R*.fastq* in queue normal to genome hg19 with site file /Users/hwick/opt/juicer/restriction_sites/hg19_HindIII.txt
(-: Created /Users/hwick/opt/juicer/work/splits and /Users/hwick/opt/juicer/work/aligned.

 Splitting files
/Users/hwick/opt/juicer/scripts/juicer.sh: line 373: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 373: bsub: command not found

/Users/hwick/opt/juicer/scripts/juicer.sh: line 395: bsub: command not found

(-: Starting job to launch other jobs once splitting is complete
/Users/hwick/opt/juicer/scripts/juicer.sh: line 453: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 471: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 512: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 546: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 594: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 654: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 668: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 683: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 705: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 727: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 741: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 774: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 793: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 803: bsub: command not found
/Users/hwick/opt/juicer/scripts/juicer.sh: line 815: bsub: command not found
(-: Finished adding all jobs... please wait while processing.

I'm not sure what this means or how it can be fixed, or where to find the results, if any. Do you have any advice? I couldn't find anything on the forum directly related to this error. Am I missing something that I need to install?

Thanks again for all your help,

~ Heather

--

Heather Wick
PhD Candidate, Human Genetics
Labs of Sarah Wheelan and Vasan Yegnasubramanian
Institute of Genetic Medicine
Johns Hopkins University School of Medicine

hwi...@jhmi.edu

[Neva Durand]

Hello Heather,

I'm not sure exactly what's going on; juicer does not delete things so that's not what's happening.  In this case, you are running the LSF scripts and not the CPU scripts, and that's what's giving you trouble.  Let's take this off the forum because I think there are some basic misunderstandings that we need to clear up.

Best

Neva